Clever Geek Handbook

What is a validator game or “how to run a proof-of-stake blockchain”

So, your team has finished the alpha version of your blockchain, and it's time to run testnet, and then mainnet. You have a real blockchain, with independent participants, a good economic model, security, you have designed governance and now it's time to try it all in business. In an ideal cryptanarchic world, you upload the genesis block to the network, the final node code and validators themselves start everything, raise all the auxiliary services and everything happens by itself. But this is in a fictional world, but in the real world, the team should prepare quite a lot of auxiliary software and various manipulations to help validators launch a stable network. About this article.







Launching consensus-based networks of the “proof-of-stake” type, where validators are determined by the votes of system token holders, is a rather specific event, because even launching traditional, centrally managed systems with dozens and hundreds of servers is not an easy task in itself, and blockchain needs to be started by efforts loyal but independent participants. And, if in a corporation, at startup, administrators have full access to all machines, logs, general monitoring, then validators will not let anyone in to their servers and, most likely, will prefer to build their own infrastructure, because it controls access to the validator's main assets - steaks voting. It is this behavior that allows you to build distributed secure networks - the independence of the cloud providers, virtual and baremetal servers used, different operating systems, all this makes attacks of such a network extremely inefficient - too many different softwares are used. For example, Ethereum uses two main node implementations, on Go and Rust, and an attack effective for one implementation does not work for another.







Therefore, all the processes of starting and operating blockchains should be organized so that any validator, or even a small group of validators, could at any time throw their computers out the window and leave, while nothing should break and the remaining validators should continue to support the work effectively network and connect new validators. When the network is launched, when one validator is in Europe, the second in South America, and the third in Asia, it is rather difficult to achieve the coordinated work of several dozen independent groups and interest them as a result.







Validators



Let's imagine launching a hypothetical modern blockchain (most of what is described is suitable for blockchains based on any modern family of blockchains: Ethereum, EOS, Polkadot, Cosmos and others, which provide proof-of-stake consensus. The main characters of such blockchains are validator teams engaged in the installation of their own independent servers, validating and producing new blocks, and receiving awards provided by the network for those who participate in consensus. to tens of validators (as can now more or less efficiently reach consensus in seconds), so the project announces the registration in which validators share public information about themselves to users by convincing them that they are going to provide quality service triggered network.







Validation is a business that allows you to accurately assess the potential income of the validator, quickly transfer capacities between projects, and if the network chosen by him is valid, the validator can develop the project as a full-fledged DAO participant and responsible person, or simply provide excellent technical service for completely transparent honestly earned money. When calculating the reward for validators, projects try to take into account the costs of validators and make a reward for blocks so that this business is profitable, but at the same time it would not allow validators to bring down the economy by filling them with money and depriving them of the rest of the network users.







The business of validators requires ensuring high fault tolerance of services, which means a high level of training of devops and developers and expensive computing resources. Even without the need to mine hashes in proof-of-work networks, the blockchain node is a large service that consumes a lot of memory, consumes a lot of computation, validates, writes to disk and transfers large amounts of data to the network. To store the transaction log and block chains for a blockchain with several thousand small transactions in a block, storage from 50 Gb or more is now required, and for blocks this should be an SSD. State database blockchains with support for smart contracts can already exceed 64Gb of RAM. Servers with the required characteristics are quite expensive, an Ethereum or EOS node can cost from 100 to 200 $ / month. Add to this the increased remuneration for the round-the-clock work of developers and devs, who during the launch period solve problems even at night, since some of the validators can easily be in the other hemisphere. However, in good times, owning a validator node can generate significant revenue (up to $ 10,000 per day for EOS).







Validation is only one of the new potential IT roles for entrepreneurs and companies, as programmers come up with more sophisticated algorithms that allow rewarding honesty and punishing deception and theft, services appear that perform the functions of publishing important data (oracles) that perform supervision (slashing deposits and punishing cheaters by publishing evidence of fraud), dispute resolution services, insurance and options services, even the garbage collection is a potentially large market in smart contract systems where it is necessary to pay for data storage.







Blockchain launch problems



The openness of the blockchain, which made it possible to freely participate in the work of a network of computers from any country and the ease of connecting any script kiddie to the network according to the instructions on GitHub, is not always an advantage. The pursuit of a new token often forces validators to “mine a new coin at the start”, in the hope of appreciation and the ability to quickly lose what they have earned. Also, this means that anyone can be your validator, even an anonymous one, you can vote for him as well as other validators (though it will be difficult for an anonymous person to collect the votes of stakeholders for himself, so we will leave the terrible tales about anonymous cryptocurrencies to politicians) . Nevertheless







The project team has a task - to somehow get into your network those who in the future are able to ensure stable operation of nodes, understand security, can quickly solve problems, cooperate with other validators and act together - the quality of that fully depends on these qualities the token itself, in which the network participants were going to invest their time and resources. Adequate founders, assessing the risks, are well aware that when launching software of such a volume, you will certainly have to face errors in the code, configuration of nodes, and that the stability of the network depends on how well developers and validators together solve such problems.







The team is ready to vote in mainnet for any validators, but just to know which ones are good? The biggest portfolio? Almost no one has it now. By team profiles on Linkedin? Experienced devops or security people will not give you any profiles on Linkedin. According to the statements in the chat, posts and help to others in the preparation phase? Good, but subjective and inaccurate.







In such circumstances, one thing remains - that solves the problems of everyone well - a game in which you can choose the best validators, but the main thing is to check the blockchain for strength and conduct a full-scale battle test of the blockchain in conditions of active use, changes in consensus, appearance and correction of errors . For the first time, this procedure was presented as a game by the guys from the Cosmos project, and this idea is undoubtedly an excellent way to prepare the network for the launch of a reliable and fault-tolerant mainnet







Game of validators



I will describe the game of validators as we designed it for the DAO.Casino (DAOBet) blockchain based on the EOS fork, which is called Haya and has a close governance mechanism - validators are selected by voting from any account in which part of the balance that votes for the validator is frozen. Any account with the main BET token on the balance sheet can vote for the selected validator with any part of its balance. The votes are summed up and the top validators are built based on the results. In different blockchains, this process is organized in different ways, and usually in this part the new blockchain differs from the parent, and I must say that in our case EOS fully justifies “OS” in its name, we really use EOS as the base operating system for deploying a modified version of the blockchain for DAOBet tasks.







I will describe individual problems and how they can be solved within the framework of the game. Imagine a network in which your server can openly attack, where in order to maintain the position of the validator, you need to continuously interact with the network, promoting your validator and making sure that it produces blocks and they are delivered to the other validators in time, otherwise, the validator will be dropped from the list.







How to choose top winners?



The main technical requirement for the game is that its results are publicly verified. This means that the results of the game: TOP winners, must be formed strictly on the basis of data that any participant can verify. In a centralized system, we could measure the “uptime” of each validator and reward those who were more online or let through a maximum of network traffic. You can collect data on CPU utilization, memory and reward those who have worked with dignity. But any such collection of metrics means the existence of a collection center, and the nodes are all independent and can behave as they want and send any data.







Therefore, the natural solution is that the winners should be determined according to the data from the blockchain, since it can be used to see which of the validators which block produced and which transactions were included in it. We called this number Validator Points (VP), and earning them is the main goal of validators in the game. In our case, the simplest, easily publicly verifiable and effective metric of the “usefulness” of the validator is VP = the number of blocks produced by the validator for a given time period.







Such a simple choice is due to the fact that governance in EOS already provides for many problems that arise, since EOS is the successor of three generations of really working blockchains with extensive experience in complex network management, and almost any validator problems with a network, processor, or disk lead to only one the problem - he signs fewer blocks, receives less pay for the work, which again leads us simply to the number of blocks to be signed - for EOS this is a great and easy option.







For other blockchains, the method of calculating Validator Points may differ, for example, for pBFT-based consensus (Tendermint / Cosmos, Aura consensus from Parity Substrate), where each block must be signed by many validators, it makes sense to read individual validator signatures, not blocks, it may make sense to consider the incomplete consensus rounds that spend the resources of other validators, in general it depends a lot on the type of consensus.







How to simulate real operating conditions



The task of the founders is to check the validators in conditions that are close to reality, while not having any centralized control. This problem can be solved with the help of a contract-faucet, which distributes equal amounts of the main token to validators and everyone. To get tokens to the balance, you need to form a transaction, and ensure that the network includes it in the block. Thus, to win, the validator needs to constantly replenish his balance with new tokens and vote for himself, promoting himself in the top. This activity creates a constant load on the network, and the parameters can be selected so that the request flow is serious enough for a full network test. Therefore, plan the contract-faucet in advance, as an important tool for launching the network, and begin to select its parameters in advance.







The request for tokens with faucet and the voting of validators still do not honestly emulate the work of the warhead, especially in extremely loaded modes. Therefore, the blockchain team will still have to write additional benchmarks anyway to load the network. A special role in this is played by specially created smart contracts in advance, which allow testing a separate subsystem. For storage testing, the contract stores random data in the blockchain, and for checking network resources, the test contract requires a large amount of input data, thereby inflating the volume of transactions - by launching the flow of such transactions at arbitrary points in time, the team simultaneously tests the stability of the code and the validity of validators.







A separate issue is updating the node code and conducting hard forks. It is required that in the event of a bug, vulnerability, collusion of malicious validators, the validators should have an action plan already worked out in the game of validators. Here you can come up with VP accrual schemes for the quick use of hard fork, for example, fine all validators who have not yet rolled up a new version of the node code, but this is difficult to implement and complicates the calculation. You can simulate the emergency situation of hard fork artificially “breaking” the blockchain on a given block. Block production stops, and as a result, those who turn on earlier and start signing blocks will benefit, so VP based on the number of signed blocks is well suited here.







How to inform participants about network status and repair errors



Despite the distrust between the validators, it’s beneficial for everyone to get up-to-date information on the network status in order to make decisions faster, so the project team raises a service for collecting and visualizing many metrics from validator servers, which allows you to see the situation simultaneously for the entire network, allowing you to quickly determine what going on. It is also beneficial for both the validators and the project that the project team quickly corrects the errors found, therefore, in addition to collecting metrics, it makes sense to immediately start collecting from the validator machines logs and error data to a machine accessible to blockchain developers. It is not beneficial for anyone to distort information, therefore these services are raised by the project team and can be trusted. It makes sense to collect system metrics from validators, and, necessarily, the most important metrics of the blockchain itself - for DAOBet, ​​this is the time of finalization and the lag of the last finalized block. Thanks to this, the team sees an increase in memory consumption on nodes when the benchmark is launched, problems of individual validators







Important points for conducting a validator game



As it turned out, if you want to officially allow validators to attack each other’s machines (unofficially they can do this), you need to formulate it separately legally as security testing, since under the laws of some countries they can punish DDoS or network attacks. Another important issue is how to reward validators. Natural prizes are project tokens, which will be transferred to mainnet, but massive distribution of tokens to anyone who was able to launch the node is also not the best option. Most likely you will have to balance between the two extreme options:









Which option to give preference is your business







There is one more point - it’s not at all a fact that dozens of validators will rush to participate in the game at your call, and of those who decide not to even install and launch the node - usually, at this stage the projects have rather poor documentation, there are errors, and developers working in time trouble answer questions not too quickly. Therefore, before launching the game it is also necessary to foresee actions if the number of validators is not needed. In this case, at the start of the game, the missing validators are launched by the project team, participate in consensus, but cannot be winners.







Conclusion



In conclusion, I tried to collect from the above a list of what you need to come up with, make and run for an effective game of validators







What you need to do to run a validator game:

develop your blockchain :)









I must say that the game of validators is a new story, and was carried out only a couple of times, so you should not take this text as a ready-made guide. There are no analogues in the modern IT business - imagine that banks, before launching a payment system, compete among themselves who is better at conducting customer transactions. Traditional approaches are unlikely to help you create large decentralized networks, so master new business models, conduct your games, identify worthy ones, reward them, and let your distributed systems work quickly and stably.







More articles:


All Articles