Grigory Petrov: working with a network in Ruby
On September 28, at Evrone's RubyRussia DevRel conference, Grigory Petrov will talk about how microservices communicate. In today's interview, Ivan Solovyov talked with Grigory about the topic of his upcoming speech and not only about that.
Tell us about yourself, what do you do in Evrone?
I’m involved in developer relations - this is something between DevRel and Technology Evangelist, a developer who can speak at conferences. Work like Robin Hood: communicate with development teams within the company, collect various interesting things from what they do, and talk about it at conferences. Ruby Russia will be one of the first conferences where I will speak on behalf of the company.
I know you as one of the organizers of various Python conferences. Tell me how you came to the world of Ruby?
I have never left the world of Ruby. One of my first lectures was Python vs Ruby. Many years ago, when I wrote in C ++, I needed high-level automation mechanisms for any testing, documentation generation. Then I wrote in Python and Ruby, and a little bit in PHP. I was looking for the best ways to solve the problems that I had.
How much will your report be borrowed from development teams within the company? I know that you wrote in many languages: Ruby, Python, PHP, TypeScript, JavaScript, C ++ and not only.
The report will be about the communication of microservices among themselves over the network, the protocols used for this and the emerging difficulties. In fact, I have a long relationship with the grid. I did Radmin, this is a network tool for remote control. The NPTV project (in which I also worked DevRel) is an interactive television that actively used the grid and controlled on Ruby. Voximplant (where I studied DevRel again) is a programmable telephony where I met VoIP. The grid is very close to me. I could make a report based on my experience, but this way I would not bring the maximum value to the guests of the conference, but I want to bear the maximum benefit. In the initial report preparation phase, I interviewed several Evrone teams. The company is engaged in custom development of complex and high-quality software, many teams are working on various projects. We called up at Zoom and talked for an hour or more. They discussed what and how they do, what difficulties, what stacks they use. My report will be half about the grid, network protocols, complexity and application, and half about how it is used in Ruby by different teams.
You have a lot of communication with developers. How specific do you think working with a network in Ruby is?
The network consists of several parts. First of all, the work of the programming language and its ecosystem directly with bytes transmitted over the network. The lowest level network stack. For example, the same JS and Node use libuv , an asynchronous model. There is a main stream, and you work with the network eventually. You have coroutines for this. You make a lot of expectations, you make a wait in order to receive data, you make a wait so that the data is sent. This single thread provides thousands and tens of thousands of queries per second.
On this foundation, frameworks are developed, whatever they are. For example, JS does not have serious frameworks for working with the grid (for which you can congratulate him!). With the exception of express.js, which is hardly a full-fledged framework. Python switched to a similar model, and the most popular Django framework remained on the previous model. Now there’s a kind of discord - a synchronous framework that tries to propagate by blocking threads, processes with a GIL hanging over it, and some new things on the side that try to work on an asynchronous model, for example Django Channels. Ruby remains in the synchronous model and propagated by processes. Therefore, there is a corresponding ecosystem, approaches and very strong positions of Rails.
What is the power of Ruby? First of all, in DSL, Domain specific language. When we talk about the grid, Ruby plays best in the field where it is the strongest. When we use GraphQL, it means that libraries for using GraphQL are everywhere. In Ruby, they will use very good DSL syntax to define the schema. And the integration between these DSL syntax and the DSL ORM syntax in ActiveRecord. This is exactly what we can expect from Ruby. At the same time, we will not have any asynchronous operations (await), we will be scaled by the processes and we will have the corresponding server requirements.
Your report stated several interaction protocols. The same JSON: API and so on. In what direction do you see further development, will we all slide into GraphQL?
A very resonant question. In my opinion, it started with the fact that the grid is slow. The applications at the beginning are simple. As a rule, all applications, including Ruby, Python, and Node, use regular HTTP endpoints for communication. Inside they use some kind of payload. Previously XML, now JSON. They are doing well for the first few months or years, as anyone is lucky. Next, the business begins to ask complex questions. For example, you need to get some users, and for each user to get a list of companies in which he works. Here the problem arises: if you just use endpoint, then it will be several tens or hundreds of requests, and the grid is leisurely: request, response, lost packets. It will be monstrously slow and much data will need to be transmitted over the network. 100 times more than the required data. Our system collapses, as large business automation systems collapse under such requests, which in 10 years grow into monsters, where a complex question from the interface can take minutes, hours. All this time, the database on the backend will work out a bunch of identical requests, and a bunch of identical requests will chase across the network. We try to solve it in different ways.
Give real examples of such problems?
Facebook was particularly distinguished, they have this problem is very acute: there is a lot of data on which they like to make complex queries. For example: show me those who commented on this post and their friends. In order not to make millions of requests, Facebook uses different options. For example, FQL, Facebook Query Language. Having collected all their expertise, they made GraphQL - a thing that allows you to make SQL-like queries on the client. But this is not SQL, because we cannot be attached to databases, but a query in terms of the backend API. You send one request and you get one (or how it goes) answer.
The second big problem is what to do if we want to get a lot of data from the backend. For example, five thousand users or a log in 10mb. It is scary to do it all with one http request-response. Because if the mesh collapses, the request will have to be repeated in its entirety, and this can last forever. Returning to Facebook: they made GraphQL, a crutch over the grid. And then other people did HTTP / 2, which solves the grid problem. HTTP / 2 makes one asynchronous connection, within which we can make many small requests. HTTP / 2 fights GraphQL if the GraphQL server does not have much magic to optimize the number of queries to the database on the backend. And in the talk, we'll talk about what Ruby offers for this magic. With HTTP / 2, GraphQL may not be needed. We can make 100 HTTP / 2 requests to our endpoint, and from the point of view of bytes this will not be a bigger overhead than if we use GraphQL. Google Protocol Buffers and gRPC approach this issue from the third end. They use binary transport protocols, mainly HTTP / 2, offer a certain scheme for api. Here they compete with the usual REST.
In practice, in most companies that use JSON, the programmer Vasya sits, who writes this JSON with his hands. Six months later, Vasya learns that the date and time can be transmitted in a hundred different ways. Horror begins! But if good developers sit in the company, they write not just JSON, but use some kind of standard. Using OpenAPI or JSON Schema, all these interesting things that compete with gRPC. This whole modern zoo solves several voiced problems. And what will happen to this zoo in the future, I cannot predict at all. But I invite developers to come and discuss this issue: what awaits us in the coming year, 3, 5, 10 and what is the disposition of forces now.
Let's talk about the future of Ruby as a programming language?
It’s hard to predict the future. I would really like to see good types in Ruby. Ruby 3 is now at the very initial stage of type implementation. I would like this syntax to be beautiful. I saw the proposals, my bald spot stood on end from them. Awful, very verbose syntax that no one will use.
Why do you think so?
I am an amateur neurophysiologist. The intuitive thinking that everyone has likes to make all sorts of strange decisions. If, for example, you need to write a lot of letters, then this is bad. These types can be mega-cool, but due to the fact that you have to write one and a half times more code, we will feel the emotion “I don’t want to”. And we are very sensitive to our emotions, so no one will use it. I really like the way the types were conveyed in Python and TypeScript: through the colon. This gives a minimal overhead. We wrote an identifier - looked. I know for sure that there will be a number, you need to put a trap. The developer writes a colon and that's it, the trap is installed. After a couple of weeks, when he accidentally passes a list or line there, the trap will work and save the developer several hours or weeks of debugging.
What else would you like to see in Ruby?
Over the past few years, I have seen a lot of async with coroutines. I really like this because async code with coroutines is easy to read. It is understandable, it allows you to cram complex things into simple syntax. This is well implemented in the latest Python and well implemented in JavaScript. I would very much like for Ruby to bring something like that ... Actually, there are fibers in Ruby. It would be cool to add something like Node so that you can write asynchronous Ruby applications using fibers or some other primitives. And under the hood, it itself would use libuv or some other primitive operating system to work with the grid.
Or would lay out something in streams. Would use something in order to competitively fulfill all these network requests, database requests, file system requests. And I would only write coroutine at the level of small pieces of code that are executed on an incoming request or timer, then give out commands and wait for their execution. Further under the hood a lot of magic, all this is done in parallel, eats up a huge Amazon car by 100% and has tens of thousands of requests per second. In the case of Go, hundreds of thousands of queries per second.
Back to the types. This will probably be the gradual introduction of types, Gradual Typing?
Gradual Typing is a mega skyrocket that was first added to Python. We made it so that types can be added a little bit. In my opinion, a whole development paradigm has arisen when in the beginning the code is written very quickly without types, and then when the developer sees that some part of the code has stabilized, types begin to add to this part of the code. It is where it has stabilized and it is necessary to set traps so that in the future nothing is broken about this stabilized one. It would be great if they did something similar for Ruby.
What question do you want to ask Yukihiro Matsumoto at the conference?
I have been studying Japanese for 4 years, and my Japanese is probably enough to say “thank you” to him. I will train!
See you at RubyRussia!
Register , the next price increase is expected after September 15, 700 participants are already with us.
And traditional thanks to the companies that support us:
Organizer - Evrone
General Partner - Toptal
Gold Partner - Gett
Silver Partners - JetBrains , Bookmate and Cashwagon
Bronze Partner - InSales
Tell us about yourself, what do you do in Evrone?
I’m involved in developer relations - this is something between DevRel and Technology Evangelist, a developer who can speak at conferences. Work like Robin Hood: communicate with development teams within the company, collect various interesting things from what they do, and talk about it at conferences. Ruby Russia will be one of the first conferences where I will speak on behalf of the company.
I know you as one of the organizers of various Python conferences. Tell me how you came to the world of Ruby?
I have never left the world of Ruby. One of my first lectures was Python vs Ruby. Many years ago, when I wrote in C ++, I needed high-level automation mechanisms for any testing, documentation generation. Then I wrote in Python and Ruby, and a little bit in PHP. I was looking for the best ways to solve the problems that I had.
How much will your report be borrowed from development teams within the company? I know that you wrote in many languages: Ruby, Python, PHP, TypeScript, JavaScript, C ++ and not only.
The report will be about the communication of microservices among themselves over the network, the protocols used for this and the emerging difficulties. In fact, I have a long relationship with the grid. I did Radmin, this is a network tool for remote control. The NPTV project (in which I also worked DevRel) is an interactive television that actively used the grid and controlled on Ruby. Voximplant (where I studied DevRel again) is a programmable telephony where I met VoIP. The grid is very close to me. I could make a report based on my experience, but this way I would not bring the maximum value to the guests of the conference, but I want to bear the maximum benefit. In the initial report preparation phase, I interviewed several Evrone teams. The company is engaged in custom development of complex and high-quality software, many teams are working on various projects. We called up at Zoom and talked for an hour or more. They discussed what and how they do, what difficulties, what stacks they use. My report will be half about the grid, network protocols, complexity and application, and half about how it is used in Ruby by different teams.
You have a lot of communication with developers. How specific do you think working with a network in Ruby is?
The network consists of several parts. First of all, the work of the programming language and its ecosystem directly with bytes transmitted over the network. The lowest level network stack. For example, the same JS and Node use libuv , an asynchronous model. There is a main stream, and you work with the network eventually. You have coroutines for this. You make a lot of expectations, you make a wait in order to receive data, you make a wait so that the data is sent. This single thread provides thousands and tens of thousands of queries per second.
On this foundation, frameworks are developed, whatever they are. For example, JS does not have serious frameworks for working with the grid (for which you can congratulate him!). With the exception of express.js, which is hardly a full-fledged framework. Python switched to a similar model, and the most popular Django framework remained on the previous model. Now there’s a kind of discord - a synchronous framework that tries to propagate by blocking threads, processes with a GIL hanging over it, and some new things on the side that try to work on an asynchronous model, for example Django Channels. Ruby remains in the synchronous model and propagated by processes. Therefore, there is a corresponding ecosystem, approaches and very strong positions of Rails.
What is the power of Ruby? First of all, in DSL, Domain specific language. When we talk about the grid, Ruby plays best in the field where it is the strongest. When we use GraphQL, it means that libraries for using GraphQL are everywhere. In Ruby, they will use very good DSL syntax to define the schema. And the integration between these DSL syntax and the DSL ORM syntax in ActiveRecord. This is exactly what we can expect from Ruby. At the same time, we will not have any asynchronous operations (await), we will be scaled by the processes and we will have the corresponding server requirements.
Your report stated several interaction protocols. The same JSON: API and so on. In what direction do you see further development, will we all slide into GraphQL?
A very resonant question. In my opinion, it started with the fact that the grid is slow. The applications at the beginning are simple. As a rule, all applications, including Ruby, Python, and Node, use regular HTTP endpoints for communication. Inside they use some kind of payload. Previously XML, now JSON. They are doing well for the first few months or years, as anyone is lucky. Next, the business begins to ask complex questions. For example, you need to get some users, and for each user to get a list of companies in which he works. Here the problem arises: if you just use endpoint, then it will be several tens or hundreds of requests, and the grid is leisurely: request, response, lost packets. It will be monstrously slow and much data will need to be transmitted over the network. 100 times more than the required data. Our system collapses, as large business automation systems collapse under such requests, which in 10 years grow into monsters, where a complex question from the interface can take minutes, hours. All this time, the database on the backend will work out a bunch of identical requests, and a bunch of identical requests will chase across the network. We try to solve it in different ways.
Give real examples of such problems?
Facebook was particularly distinguished, they have this problem is very acute: there is a lot of data on which they like to make complex queries. For example: show me those who commented on this post and their friends. In order not to make millions of requests, Facebook uses different options. For example, FQL, Facebook Query Language. Having collected all their expertise, they made GraphQL - a thing that allows you to make SQL-like queries on the client. But this is not SQL, because we cannot be attached to databases, but a query in terms of the backend API. You send one request and you get one (or how it goes) answer.
The second big problem is what to do if we want to get a lot of data from the backend. For example, five thousand users or a log in 10mb. It is scary to do it all with one http request-response. Because if the mesh collapses, the request will have to be repeated in its entirety, and this can last forever. Returning to Facebook: they made GraphQL, a crutch over the grid. And then other people did HTTP / 2, which solves the grid problem. HTTP / 2 makes one asynchronous connection, within which we can make many small requests. HTTP / 2 fights GraphQL if the GraphQL server does not have much magic to optimize the number of queries to the database on the backend. And in the talk, we'll talk about what Ruby offers for this magic. With HTTP / 2, GraphQL may not be needed. We can make 100 HTTP / 2 requests to our endpoint, and from the point of view of bytes this will not be a bigger overhead than if we use GraphQL. Google Protocol Buffers and gRPC approach this issue from the third end. They use binary transport protocols, mainly HTTP / 2, offer a certain scheme for api. Here they compete with the usual REST.
In practice, in most companies that use JSON, the programmer Vasya sits, who writes this JSON with his hands. Six months later, Vasya learns that the date and time can be transmitted in a hundred different ways. Horror begins! But if good developers sit in the company, they write not just JSON, but use some kind of standard. Using OpenAPI or JSON Schema, all these interesting things that compete with gRPC. This whole modern zoo solves several voiced problems. And what will happen to this zoo in the future, I cannot predict at all. But I invite developers to come and discuss this issue: what awaits us in the coming year, 3, 5, 10 and what is the disposition of forces now.
Let's talk about the future of Ruby as a programming language?
It’s hard to predict the future. I would really like to see good types in Ruby. Ruby 3 is now at the very initial stage of type implementation. I would like this syntax to be beautiful. I saw the proposals, my bald spot stood on end from them. Awful, very verbose syntax that no one will use.
Why do you think so?
I am an amateur neurophysiologist. The intuitive thinking that everyone has likes to make all sorts of strange decisions. If, for example, you need to write a lot of letters, then this is bad. These types can be mega-cool, but due to the fact that you have to write one and a half times more code, we will feel the emotion “I don’t want to”. And we are very sensitive to our emotions, so no one will use it. I really like the way the types were conveyed in Python and TypeScript: through the colon. This gives a minimal overhead. We wrote an identifier - looked. I know for sure that there will be a number, you need to put a trap. The developer writes a colon and that's it, the trap is installed. After a couple of weeks, when he accidentally passes a list or line there, the trap will work and save the developer several hours or weeks of debugging.
What else would you like to see in Ruby?
Over the past few years, I have seen a lot of async with coroutines. I really like this because async code with coroutines is easy to read. It is understandable, it allows you to cram complex things into simple syntax. This is well implemented in the latest Python and well implemented in JavaScript. I would very much like for Ruby to bring something like that ... Actually, there are fibers in Ruby. It would be cool to add something like Node so that you can write asynchronous Ruby applications using fibers or some other primitives. And under the hood, it itself would use libuv or some other primitive operating system to work with the grid.
Or would lay out something in streams. Would use something in order to competitively fulfill all these network requests, database requests, file system requests. And I would only write coroutine at the level of small pieces of code that are executed on an incoming request or timer, then give out commands and wait for their execution. Further under the hood a lot of magic, all this is done in parallel, eats up a huge Amazon car by 100% and has tens of thousands of requests per second. In the case of Go, hundreds of thousands of queries per second.
Back to the types. This will probably be the gradual introduction of types, Gradual Typing?
Gradual Typing is a mega skyrocket that was first added to Python. We made it so that types can be added a little bit. In my opinion, a whole development paradigm has arisen when in the beginning the code is written very quickly without types, and then when the developer sees that some part of the code has stabilized, types begin to add to this part of the code. It is where it has stabilized and it is necessary to set traps so that in the future nothing is broken about this stabilized one. It would be great if they did something similar for Ruby.
What question do you want to ask Yukihiro Matsumoto at the conference?
I have been studying Japanese for 4 years, and my Japanese is probably enough to say “thank you” to him. I will train!
See you at RubyRussia!
Register , the next price increase is expected after September 15, 700 participants are already with us.
And traditional thanks to the companies that support us:
Organizer - Evrone
General Partner - Toptal
Gold Partner - Gett
Silver Partners - JetBrains , Bookmate and Cashwagon
Bronze Partner - InSales
All Articles