Applied technology on the ruins of blockchain fever or the practical benefits of resource allocation
In recent years, news feeds have flooded with messages about a new type of distributed computing networks that solve (or rather try to solve) the most diverse tasks - make the city smart, save the world from copyright infringers, or vice versa, secretly transfer information or resources, escape from - under the control of the state in a particular area. Regardless of the sphere, they all have a number of common features, due to the fact that the fuel for their growth was algorithms and techniques that came to the masses during the recent boom of cryptocurrencies and related technologies. Probably every third article on specialized resources at that time in the title had the word “blockchain” - discussion of new software solutions and economic models for some time became the dominant trend, against which other areas of application of distributed computing systems were relegated to the background.
At the same time, visionaries and professionals saw the main essence of the phenomenon: mass distributed computing related to building networks of a large number of disparate and heterogeneous participants reached a new level of development. It is enough to throw hype topics out of your head and look at the subject from the other side: all these networks, assembled from huge pools, which consist of thousands of separate heterogeneous participants, did not appear on their own. Crypto movement enthusiasts were able to solve in a new way the complex problems of data synchronization and the allocation of resources and tasks, which allowed us to put together a similar mass of equipment and create a new ecosystem designed to solve one narrowly targeted problem.
Of course, this did not pass by the teams and communities involved in the development of free distributed computing, and new projects were not long in coming.
However, despite a serious increase in the amount of available information about developments in the field of networking and equipment, the creators of promising systems will have to solve serious problems.
The direction can be right, it can lead to a dead end - you can’t escape from it, the centralized deliveries of clairvoyants to the IT community are still late. But the choice should be made so as not to fall into the traditional trap that the team takes too wide a field and from the start tries to create another non-specialized project of distributed computing with a wide profile. It seems that the front of the work is not so terrible, for the most part it is just necessary to apply the existing developments: unite the nodes into a network, adapt the algorithms for determining topologies, exchange data and control their consistency, introduce methods for ranking nodes and finding consensus, and, of course, just create your own query language and the entire language and computing environment. The idea of a universal mechanism is very tempting and constantly pops up in one sphere or another, but one of the three results is still obtained: the created solution either turns out to be actually a limited prototype with a bunch of suspended “ToDo” in the backlog, or it becomes an unworkable monster, ready to drag anyone who touches the fetid “Turing quagmire”, or simply dies safely from the fact that the swan, cancer and pike, pulling in an incomprehensible direction, are torn to pieces.
We will not repeat stupid mistakes and choose a direction that has an understandable range of tasks and is well suited to the distributed computing model. You can understand people who are trying to do everything at once - of course, there are plenty to choose from. And much looks extremely interesting both from the point of R&D and development, and from the point of view of the economy. Using a distributed network, you can:
In order not to get carried away by compiling a list of interesting things that parallel well, we will choose distributed rendering as our further topic.
Distributed rendering in itself is, of course, never a new phenomenon. existing render toolkits have long supported the distribution of load across different machines, without which it would be rather sad to live in the twenty-first century. However, you should not think that the topic has been driven up and down, and there is nothing to do there - we will consider a separate urgent problem: creating a tool for creating a render network.
The rendering network we have is the union of nodes that need to perform rendering tasks, with nodes that have free computing resources to handle rendering. Resource owners will connect their stations to the rendering network in order to receive and execute rendering tasks using one of the network-supported rendering engines. At the same time, task providers will work with the network as with a cloud that independently performs resource allocation, control of correct execution, risk management and other problems.
Thus, we will consider the creation of a framework that should support integration with a set of popular render engines and contain components that provide tools for organizing a network of heterogeneous nodes and controlling the flow of tasks.
The economic model for the existence of such a network is not of fundamental importance, therefore, for the initial one we will take a scheme similar to that used in calculations in cryptocurrency networks - resource consumers will send tokens to suppliers that perform rendering work. It is much more interesting to understand what properties a framework should have, for which we will consider the main scenario for the interaction of network participants.
There are three aspects of interaction in a network: a resource provider, a task provider, and a network operator (aka a control center, a network, etc. throughout the text).
The network operator provides the resource provider with a client application or an operating system image with a comprehensive set of software that he will install on the machine whose resources he wants to provide, and a personal account accessible through the web interface, allowing him to set access parameters to the resource and remotely control his server landscape: control the parameters of iron, perform remote configuration, reboot.
When connecting a new node, the network management system analyzes the equipment and the specified access parameters, ranks it, assigns a certain rating, and places it in the resource register. In the future, in order to manage risk, the parameters of the activity of the node will be analyzed, and the rating of the node will be adjusted to ensure the stability of the network. Nobody will be pleased if their scene is sent to render on powerful, but often overheated cards?
A user who needs to render a scene can go in two ways: upload the scene to the network repository via the web interface or use the plug-in to connect his simulation package or the installed renderer to the network. In this case, a smart contract is initiated between the user and the network, the standard condition for the completion of which is the generation of the scene calculation result by the network. The user can monitor the progress of the task and manage its parameters through the web interface of his personal account.
The task arrives at the server, where the scene volume and the number of resources requested by the task initiator are analyzed, after which the total volume is decomposed into parts adapted for calculation on the number and type of resources allocated by the network. The general idea is that visualization can be broken down into many small tasks. Engines take advantage of this by distributing these tasks among multiple resource providers. The simplest way is to render small parts of a scene called segments. When each segment is ready, the local task is considered completed, the resource proceeds to the next unsolved one.
Thus, for the renderer there is no difference as such whether the calculations are performed on one machine or on a grid from many separate computing stations. Distributed rendering simply adds more cores to the pool of resources used for the task. Through the network, he receives all the data necessary for rendering a segment, calculates it, sends this segment back, and proceeds to the next task. Before entering the general pool of the network, each segment receives a set of meta-information, which allows the executing nodes to choose the most suitable computing tasks for them.
The tasks of segmentation and distribution of computations should be solved not only from the point of view of optimizing the execution time, but also from the point of view of the optimal use of resources and energy saving, since the economic efficiency of the network depends on this. In the event of an unsuccessful decision, it will be more expedient to put the miner on or turn it off so that it does not make noise and does not waste electricity.
But back to the process. When a task is received between the pool and the node, a smart contract is also formed that is executed when the task result is correctly calculated. Based on the results of the contract, the node can receive a reward in one form or another.
The control center monitors the process of completing the task, collecting the results of the calculations, sending the incorrect ones for reprocessing and ranking the queue, monitoring the standard time for the task to be completed (so that it doesn't happen that the last segment is not taken into operation by any node).
The calculation results go through the compositing stage, after which the user receives the rendering results, and the network can receive a reward.
Thus, the functional composition of the landscape framework, intended for building distributed rendering systems, emerges:
At the same time, visionaries and professionals saw the main essence of the phenomenon: mass distributed computing related to building networks of a large number of disparate and heterogeneous participants reached a new level of development. It is enough to throw hype topics out of your head and look at the subject from the other side: all these networks, assembled from huge pools, which consist of thousands of separate heterogeneous participants, did not appear on their own. Crypto movement enthusiasts were able to solve in a new way the complex problems of data synchronization and the allocation of resources and tasks, which allowed us to put together a similar mass of equipment and create a new ecosystem designed to solve one narrowly targeted problem.
Of course, this did not pass by the teams and communities involved in the development of free distributed computing, and new projects were not long in coming.
However, despite a serious increase in the amount of available information about developments in the field of networking and equipment, the creators of promising systems will have to solve serious problems.
The first of them, however strange it may sound, is the problem of choosing a direction
The direction can be right, it can lead to a dead end - you can’t escape from it, the centralized deliveries of clairvoyants to the IT community are still late. But the choice should be made so as not to fall into the traditional trap that the team takes too wide a field and from the start tries to create another non-specialized project of distributed computing with a wide profile. It seems that the front of the work is not so terrible, for the most part it is just necessary to apply the existing developments: unite the nodes into a network, adapt the algorithms for determining topologies, exchange data and control their consistency, introduce methods for ranking nodes and finding consensus, and, of course, just create your own query language and the entire language and computing environment. The idea of a universal mechanism is very tempting and constantly pops up in one sphere or another, but one of the three results is still obtained: the created solution either turns out to be actually a limited prototype with a bunch of suspended “ToDo” in the backlog, or it becomes an unworkable monster, ready to drag anyone who touches the fetid “Turing quagmire”, or simply dies safely from the fact that the swan, cancer and pike, pulling in an incomprehensible direction, are torn to pieces.
We will not repeat stupid mistakes and choose a direction that has an understandable range of tasks and is well suited to the distributed computing model. You can understand people who are trying to do everything at once - of course, there are plenty to choose from. And much looks extremely interesting both from the point of R&D and development, and from the point of view of the economy. Using a distributed network, you can:
- Train neural networks
- Process signal streams
- Calculate protein structure
- Render 3D scenes
- Simulate hydrodynamics
- Test trading strategies for stock exchanges
In order not to get carried away by compiling a list of interesting things that parallel well, we will choose distributed rendering as our further topic.
Distributed rendering in itself is, of course, never a new phenomenon. existing render toolkits have long supported the distribution of load across different machines, without which it would be rather sad to live in the twenty-first century. However, you should not think that the topic has been driven up and down, and there is nothing to do there - we will consider a separate urgent problem: creating a tool for creating a render network.
The rendering network we have is the union of nodes that need to perform rendering tasks, with nodes that have free computing resources to handle rendering. Resource owners will connect their stations to the rendering network in order to receive and execute rendering tasks using one of the network-supported rendering engines. At the same time, task providers will work with the network as with a cloud that independently performs resource allocation, control of correct execution, risk management and other problems.
Thus, we will consider the creation of a framework that should support integration with a set of popular render engines and contain components that provide tools for organizing a network of heterogeneous nodes and controlling the flow of tasks.
The economic model for the existence of such a network is not of fundamental importance, therefore, for the initial one we will take a scheme similar to that used in calculations in cryptocurrency networks - resource consumers will send tokens to suppliers that perform rendering work. It is much more interesting to understand what properties a framework should have, for which we will consider the main scenario for the interaction of network participants.
There are three aspects of interaction in a network: a resource provider, a task provider, and a network operator (aka a control center, a network, etc. throughout the text).
The network operator provides the resource provider with a client application or an operating system image with a comprehensive set of software that he will install on the machine whose resources he wants to provide, and a personal account accessible through the web interface, allowing him to set access parameters to the resource and remotely control his server landscape: control the parameters of iron, perform remote configuration, reboot.
When connecting a new node, the network management system analyzes the equipment and the specified access parameters, ranks it, assigns a certain rating, and places it in the resource register. In the future, in order to manage risk, the parameters of the activity of the node will be analyzed, and the rating of the node will be adjusted to ensure the stability of the network. Nobody will be pleased if their scene is sent to render on powerful, but often overheated cards?
A user who needs to render a scene can go in two ways: upload the scene to the network repository via the web interface or use the plug-in to connect his simulation package or the installed renderer to the network. In this case, a smart contract is initiated between the user and the network, the standard condition for the completion of which is the generation of the scene calculation result by the network. The user can monitor the progress of the task and manage its parameters through the web interface of his personal account.
The task arrives at the server, where the scene volume and the number of resources requested by the task initiator are analyzed, after which the total volume is decomposed into parts adapted for calculation on the number and type of resources allocated by the network. The general idea is that visualization can be broken down into many small tasks. Engines take advantage of this by distributing these tasks among multiple resource providers. The simplest way is to render small parts of a scene called segments. When each segment is ready, the local task is considered completed, the resource proceeds to the next unsolved one.
Thus, for the renderer there is no difference as such whether the calculations are performed on one machine or on a grid from many separate computing stations. Distributed rendering simply adds more cores to the pool of resources used for the task. Through the network, he receives all the data necessary for rendering a segment, calculates it, sends this segment back, and proceeds to the next task. Before entering the general pool of the network, each segment receives a set of meta-information, which allows the executing nodes to choose the most suitable computing tasks for them.
The tasks of segmentation and distribution of computations should be solved not only from the point of view of optimizing the execution time, but also from the point of view of the optimal use of resources and energy saving, since the economic efficiency of the network depends on this. In the event of an unsuccessful decision, it will be more expedient to put the miner on or turn it off so that it does not make noise and does not waste electricity.
But back to the process. When a task is received between the pool and the node, a smart contract is also formed that is executed when the task result is correctly calculated. Based on the results of the contract, the node can receive a reward in one form or another.
The control center monitors the process of completing the task, collecting the results of the calculations, sending the incorrect ones for reprocessing and ranking the queue, monitoring the standard time for the task to be completed (so that it doesn't happen that the last segment is not taken into operation by any node).
The calculation results go through the compositing stage, after which the user receives the rendering results, and the network can receive a reward.
Thus, the functional composition of the landscape framework, intended for building distributed rendering systems, emerges:
- Web dashboards
- A set of software for installation on nodes
- By management system:
- Access control subsystem
- The decomposition subsystem of rendering tasks
- Task distribution subsystem
- Compositing Subsystem
- Server landscape and network topology management subsystem
- Logging and Auditing Subsystem
- Learning Expert Subsystem
- Rest API or other interface for external developers
What do you think? What questions does the topic raise and what answers are you interested in?
All Articles