Public test: solution for privacy and scalability in Ethereum
Blockchain is an innovative technology that promises to improve many areas of human life. It transfers real processes and products to digital space, ensures the speed and reliability of financial transactions, reduces their cost, and also allows you to create modern DAPP applications using smart contracts in decentralized networks.
Given the many advantages and diverse uses of the blockchain, it may seem strange that this promising technology has not yet penetrated all sectors. The problem is that modern decentralized blockchains lack scalability. Ethereum processes about 20 transactions per second, which is not enough to meet the needs of today's dynamic business. At the same time, companies using blockchain technology do not dare to abandon Ethereum because of its high degree of protection against hacking and network failures.
To ensure decentralization, security and scalability on the blockchain, thus solving the Scalability Trilemma, the Opporty development team created Plasma Cash - a child chain consisting of a smart contract and a private network based on Node.js, periodically transferring its state to the root chain ( Ethereum).
1. The user calls the function of the smart contract `deposit`, transferring the amount in ETH into it, which he wants to put in the Plasma Cash token. The smart contract function creates a token and generates an event about it.
2. Plasma Cash nodes subscribed to the events of the smart contract receive an event about the creation of a deposit and add a transaction about the creation of a token to the pool.
3. Periodically, special Plasma Cash nodes take all transactions from the pool (up to 1 million) and form a block from them, calculate the Merkle tree and, accordingly, the hash. This block is sent to other nodes for verification. The nodes check whether the Merkle hash is valid, whether the transactions are valid (for example, whether the sender of the token is its owner). After verifying the block, the node calls the `submitBlock` function of the smart contract, which stores the number and Merkle hash of the block in the trace chain. A smart contract generates an event about the successful addition of a block. Transactions are deleted from the pool.
4. The nodes that received the event about the submission of the block begin to apply the transactions that were added to the block.
5. At some point, the owner (or non-owner) of the token wants to withdraw it from Plasma Cash. To do this, he calls the `startExit` function, passing information about the last 2 transactions on the token into it, which confirm that he is the owner of the token. The smart contract, using the Merkle hash, checks for transactions in blocks and sends a token to the output, which will happen in two weeks.
6. If the token withdrawal operation occurred with violations (the token was spent after the start of the withdrawal procedure or the token was already a stranger to the withdrawal), the owner of the token can refute the withdrawal within two weeks.
1. The root chain does not know anything about transactions that are generated and forwarded inside the child chain. Information remains on who started and withdrawed ETH to / from Plasma Cash.
2. The child chain allows you to organize anonymous transactions using zk-SNARKs.
When developing Plasma Cash, we tested the speed of the system and got the following results:
Tests were conducted on the following 3 servers:
1. Intel Core i7-6700 Quad-Core Skylake incl. NVMe SSD - 512 GB, 64 GB DDR4 RAM
3 validated Plasma Cash nodes were raised.
2. AMD Ryzen 7 1700X Octa-Core “Summit Ridge” (Zen), SATA SSD - 500 GB, 64 GB DDR4 RAM
The Ropsten testnet ETH node was raised.
3 validated Plasma Cash nodes were raised.
3. Intel Core i9-9900K Octa-Core incl. NVMe SSD - 1 TB, 64 GB DDR4 RAM
1 Submit Plasma Cash node was raised.
3 validated Plasma Cash nodes were raised.
A test was launched to add transactions to the Plasma Cash network.
Total: 10 Plasma Cash nodes in a private network.
There is a limit of 1 million transactions per block. Therefore, 1 million transactions fall into 2 blocks (since the system manages to take part of the transactions and submits while they are sent).
Initial state: last block # 7; 1 million transactions and tokens are stored in the database.
00:00 - start the transaction generation script
01:37 - 1 million transactions were created and sending started in the node
01:46 - the submit node took 240k transactions from the pool and forms block # 8. We also see that 320k transactions are added to the pool in 10 seconds
01:58 - block # 8 is signed and sent for validation
02:03 - block # 8 is validated and the `submitBlock` function of the smart contract with the Merkle hash and block number is called
02:10 - the demo script finished working, which sent 1 million transactions in 32 seconds
02:33 - the nodes began to receive information that block # 8 was added to the root chain, and began to perform 240k transactions
02:40 - 240k transactions were deleted from the pool, which are already in block # 8
02:56 - submit node took the remaining 760k transactions from the pool and began to calculate the Merkle hash and sign block # 9
03:20 - all nodes contain 1mln 240k transactions and tokens
03:35 - block # 9 is signed and sent for validation to other nodes
03:41 - a network error has occurred
04:40 - the timeout for waiting for the validation of block # 9 has stopped
04:54 - submit node took the remaining 760k transactions from the pool and began to calculate the Merkle hash and sign block # 9
05:32 - block # 9 is signed and sent for validation to other nodes
05:53 - block # 9 is validated and sent to the root chain
06:17 - nodes began to receive information that block # 9 was added to the root chain and began to perform 760k transactions
06:47 - the pool cleared of transactions that are in block # 9
09:06 - all nodes contain 2 million transactions and tokens
There is a limit of 350k per block. As a result, we have 3 blocks.
Initial state: last block # 9; 2 million transactions and tokens stored in the database
00:00 - transaction generation script is already running
00:44 - 1 million transactions were created and sending to the node began
00:56 - the submit node took 320k transactions from the pool and forms block # 10. We also see that 320k transactions are added to the pool in 10 seconds
01:12 - block # 10 is signed and sent to other nodes for validation
01:18 - the demo script finished working, which sent 1 million transactions in 34 seconds
01:20 - block # 10 is validated and sent to the root chain
01:51 - all nodes received information from the root chain that block # 10 has been added, and they begin to apply 320k transactions
02:01 - the pool cleared for 320k transactions that were added to block # 10
02:15 - submit node took 350k transactions from the pool and forms block # 11
02:34 - block # 11 is signed and sent to other nodes for validation
02:51 - block # 11 is validated and sent to the root chain
02:55 - the last node executed transactions from block # 10
10:59 - a very long time in the root chain a transaction was carried out with the submission of block # 9, but it completed and all the nodes received information about this and started to execute 350k transactions
11:05 - the pool was cleared for 320k transactions that were added to block # 11
12:10 - all nodes contain 1 million 670k transactions and tokens
12:17 - submit node took 330k transactions from the pool and forms block # 12
12:32 - block # 12 is signed and sent to other nodes for validation
12:39 - block # 12 is validated and sent to the root chain
13:44 - all nodes received information from the root chain that block # 12 has been added and are starting to apply 330k transactions
14:50 - all nodes contain 2 million transactions and tokens
In the first and second servers, one validating node was replaced with a submit node.
Initial state: last block # 84; 0 transactions and tokens are stored in the database
00:00 - 3 scripts are launched that generate and send 1 million transactions
01:38 - 1 million transactions were created and sending to submit node # 3 began
01:50 - submit node # 3 took 330k transactions from the pool and forms block # 85 (f21). We also see that 350k transactions are added to the pool in 10 seconds
01:53 - 1 million transactions were created and sending to submit node # 1 began
01:50 - submit node # 3 took 330k transactions from the pool and forms block # 85 (f21). We also see that 350k transactions are added to the pool in 10 seconds
02:01 - submit node # 1 took 250k transactions from the pool and forms block # 85 (65e)
02:06 - block # 85 (f21) is signed and sent to other nodes for validation
02:08 - the demo script of server # 3 has finished working, which sent 1 million transactions in 30 seconds
02:14 - block # 85 (f21) is validated and sent to the root chain
02:19 - block # 85 (65e) is signed and sent to other nodes for validation
02:22 - 1 million transactions were created and sending to submit node # 2 began
02:27 - block # 85 (65e) is validated and sent to the root chain
02:29 - submit node # 2 took from the pool 111855 transactions and forms block # 85 (256).
02:36 - block # 85 (256) is signed and sent to other nodes for validation
02:36 - the demo script of server # 1 finished working, which sent 1mln transactions in 42.5 seconds
02:38 - block # 85 (256) is validated and sent to the root chain
03:08 - the server script script # 2 finished working, which sent 1 million transactions in 47 seconds
03:38 - all nodes received information from the root chain that the blocks # 85 (f21), # 86 (65e), # 87 (256) were added and begin to apply 330k, 250k, 111855 transactions
03:49 - the pool was cleared at 330k, 250k, 111855 transactions that were added to blocks # 85 (f21), # 86 (65e), # 87 (256)
03:59 - submit node # 1 took from the pool 888145 transactions and forms block # 88 (214), submit node # 2 took from the pool 750k transactions and forms block # 88 (50a), submit node # 3 took from the pool 670k transactions and forms block # 88 (d3b)
04:44 - block # 88 (d3b) is signed and sent to other nodes for validation
04:58 - block # 88 (214) is signed and sent to other nodes for validation
05:11 - block # 88 (50a) is signed and sent to other nodes for validation
05:11 - block # 85 (d3b) is validated and sent to the root chain
05:36 - block # 85 (214) is validated and sent to the root chain
05:43 - all nodes received information from the root chain that blocks # 88 (d3b), # 89 (214) were added and begin to apply 670k, 750k transactions
06:50 - due to a disconnection, block # 85 (50a) was not validated
06:55 - submit node # 2 took 888145 transactions from the pool and forms block # 90 (50a)
08:14 - block # 90 (50a) is signed and sent to other nodes for validation
09:04 - block # 90 (50a) is validated and sent to the root chain
11:23 - all nodes received information from the root chain that block # 90 (50a) was added, and 888145 transactions began to be applied. At the same time, server # 3 has long applied transactions from blocks # 88 (d3b), # 89 (214)
12:11 - all pools are empty
13:41 - all server nodes # 3 contain 3 million transactions and tokens
14:35 - all server nodes # 1 contain 3 million transactions and tokens
19:24 - all server nodes # 2 contain 3 million transactions and tokens
During the development of Plasma Cash, we encountered the following problems, which we gradually solved and are solving:
1. The conflict of interaction of various functions of the system. For example, the function of adding transactions to the pool blocked the submission and validation of blocks, and vice versa, which led to a drawdown in speed.
2. It was not immediately clear how to send a huge number of transactions and at the same time minimize the cost of data transfer.
3. It was not clear how and where to store data in order to achieve high results.
4. It was not clear how to organize a network between nodes, since the block size with 1 million transactions takes about 100 MB.
5. Working in single-threaded mode breaks the connection between nodes when long calculations occur (for example, building a Merkle tree and calculating its hash).
The first version of the Plasma Cash node was a kind of combiner that could do everything at the same time: accept transactions, submit and validate blocks, provide an API for accessing data. Since NodeJS was initially single-threaded, the heavy Merkle tree calculation function blocked the add transaction function. We saw two options for solving this problem:
1. Run several NodeJS processes, each of which performs certain functions.
2. Use worker_threads and put the execution of the code into threads.
As a result, we took advantage of both options at the same time: logically divided one node into 3 parts, which can work separately, but at the same time synchronously
1. Submit a node that accepts transactions to the pool and creates blocks.
2. Validating node that verifies the validity of nodes.
3. Node API - provides an API for accessing data.
At the same time, you can connect to each node through a unix socket using cli.
Heavy operations, such as the calculation of the Merkle tree, we carried out in a separate stream.
Thus, we achieved the normal operation of all Plasma Cash functions simultaneously and without failures.
As soon as the system functionally worked, we began to test the speed and, unfortunately, got unsatisfactory results: 5,000 transactions per second and up to 50,000 transactions in a block. I had to find out what was implemented incorrectly.
To begin with, we started testing the mechanism of communication with Plasma Cash in order to find out the peak capability of the system. Earlier we wrote that the Plasma Cash node provides a unix socket interface. It was originally textual. json objects were sent using `JSON.parse ()` and `JSON.stringify ()`.
We measured the transfer speed of such objects and received ~ 130k per second. We tried to replace the standard functions with json, but the performance did not improve. There must be a V8 engine well optimized for these operations.
Work with transactions, tokens, blocks was carried out through classes. When creating such classes, performance slipped by 2 times, which indicates: OOP is not suitable for us. I had to rewrite everything on a purely functional approach.
Initially, Redis was chosen for data storage as one of the most productive solutions that satisfies our requirement: key-value storage, work with hash tables, and many. We launched redis-benchmark and got ~ 80k operations per second in 1 pipelining mode.
For high performance, we tuned Redis more finely:
In Redis, a pool is a hash table, since we need the ability to receive all transactions in one request and delete transactions one by one. We tried to use a regular list, but it works more slowly when unloading the entire list.
Using the standard NodeJS library, Redis libraries achieved 18k transactions per second performance. Speed dropped 9 times.
Since the benchmark showed us the possibilities clearly 5 times more, they began to optimize. We changed the library to ioredis and got a performance of 25k per second. We added transactions one by one using the `hset` command. Thus, we generated a lot of queries in Redis. There was an idea to merge transactions into bundles and send them with one hmset command. The result is 32k per second.
For several reasons, which will be described below, we work with data using `Buffer` and, as it turned out, if you translate it into text (` buffer.toString ('hex') `) before writing, you can get additional performance. Thus, the speed was increased to 35k per second. At the moment, we decided to suspend further optimization.
We had to switch to a binary protocol because:
1. The system often calculates hashes, signatures, etc., and for this it needs data in `Buffer.
2. When transferring between services, binary data weighs less than text. For example, when sending a block with 1 million transactions, the data in the text can occupy more than 300 megabytes.
3. Continuous data conversion affects performance.
Therefore, we took as our basis our own binary protocol for storing and transmitting data, developed on the basis of the remarkable library `binary-data`.
As a result, we have the following data structures:
By the usual commands `BD.encode (block, Protocol) .slice ();` and `BD.decode (buffer, Protocol)` we convert the data to `Buffer` to save it to Redis or transfer it to another node and retrieve the data back.
We also have 2 binary protocols for transferring data between services:
- Protocol for interacting with Plasma Node via unix socket
Where:
- Protocol of interaction between nodes
Where:
Since we typed the data beforehand, the final system is much faster than the `rlp` library from Ethereum. Unfortunately, we have not yet been able to refuse it, since it is necessary to finalize the smart contract, which we plan to do in the future.
If we managed to achieve a speed of 35,000 transactions per second, we also need to process them in optimal time. Since the approximate formation time of the block takes 30 seconds, we need to include 1,000,000 transactions in the block, which means sending more than 100 mb of data.
Initially, we used the `ethereumjs-devp2p` library to communicate nodes, but it could not cope with so much data. As a result, we used the `ws` library and set up binary data forwarding via websocket. Of course, we also encountered problems when sending large data packets, but we divided them into chunks and now there are no such problems.
Also, the formation of the Merkle tree and the calculation of the hash of 1,000,000 transactions requires about 10 seconds of continuous calculation. During this time, the connection to all nodes manages to break. It was decided to transfer this calculation to a separate thread.
In fact, our findings are not new, but for some reason, many experts forget about them during development.
We invite you to visit the GitHub project: https://github.com/opporty-com/Plasma-Cash/tree/new-version
The article was co-written by Alexander Nashivan , senior developer of Clever Solution Inc.
Given the many advantages and diverse uses of the blockchain, it may seem strange that this promising technology has not yet penetrated all sectors. The problem is that modern decentralized blockchains lack scalability. Ethereum processes about 20 transactions per second, which is not enough to meet the needs of today's dynamic business. At the same time, companies using blockchain technology do not dare to abandon Ethereum because of its high degree of protection against hacking and network failures.
To ensure decentralization, security and scalability on the blockchain, thus solving the Scalability Trilemma, the Opporty development team created Plasma Cash - a child chain consisting of a smart contract and a private network based on Node.js, periodically transferring its state to the root chain ( Ethereum).
Key processes at Plasma Cash
1. The user calls the function of the smart contract `deposit`, transferring the amount in ETH into it, which he wants to put in the Plasma Cash token. The smart contract function creates a token and generates an event about it.
2. Plasma Cash nodes subscribed to the events of the smart contract receive an event about the creation of a deposit and add a transaction about the creation of a token to the pool.
3. Periodically, special Plasma Cash nodes take all transactions from the pool (up to 1 million) and form a block from them, calculate the Merkle tree and, accordingly, the hash. This block is sent to other nodes for verification. The nodes check whether the Merkle hash is valid, whether the transactions are valid (for example, whether the sender of the token is its owner). After verifying the block, the node calls the `submitBlock` function of the smart contract, which stores the number and Merkle hash of the block in the trace chain. A smart contract generates an event about the successful addition of a block. Transactions are deleted from the pool.
4. The nodes that received the event about the submission of the block begin to apply the transactions that were added to the block.
5. At some point, the owner (or non-owner) of the token wants to withdraw it from Plasma Cash. To do this, he calls the `startExit` function, passing information about the last 2 transactions on the token into it, which confirm that he is the owner of the token. The smart contract, using the Merkle hash, checks for transactions in blocks and sends a token to the output, which will happen in two weeks.
6. If the token withdrawal operation occurred with violations (the token was spent after the start of the withdrawal procedure or the token was already a stranger to the withdrawal), the owner of the token can refute the withdrawal within two weeks.
Privacy is achieved in two ways.
1. The root chain does not know anything about transactions that are generated and forwarded inside the child chain. Information remains on who started and withdrawed ETH to / from Plasma Cash.
2. The child chain allows you to organize anonymous transactions using zk-SNARKs.
Technological stack
- NodeJS
- Redis
- Ethereum
- Soild
Testing
When developing Plasma Cash, we tested the speed of the system and got the following results:
- up to 35,000 transactions per second are added to the pool;
- Up to 1,000,000 transactions can be stored in the block.
Tests were conducted on the following 3 servers:
1. Intel Core i7-6700 Quad-Core Skylake incl. NVMe SSD - 512 GB, 64 GB DDR4 RAM
3 validated Plasma Cash nodes were raised.
2. AMD Ryzen 7 1700X Octa-Core “Summit Ridge” (Zen), SATA SSD - 500 GB, 64 GB DDR4 RAM
The Ropsten testnet ETH node was raised.
3 validated Plasma Cash nodes were raised.
3. Intel Core i9-9900K Octa-Core incl. NVMe SSD - 1 TB, 64 GB DDR4 RAM
1 Submit Plasma Cash node was raised.
3 validated Plasma Cash nodes were raised.
A test was launched to add transactions to the Plasma Cash network.
Total: 10 Plasma Cash nodes in a private network.
Test 1
There is a limit of 1 million transactions per block. Therefore, 1 million transactions fall into 2 blocks (since the system manages to take part of the transactions and submits while they are sent).
Initial state: last block # 7; 1 million transactions and tokens are stored in the database.
00:00 - start the transaction generation script
01:37 - 1 million transactions were created and sending started in the node
01:46 - the submit node took 240k transactions from the pool and forms block # 8. We also see that 320k transactions are added to the pool in 10 seconds
01:58 - block # 8 is signed and sent for validation
02:03 - block # 8 is validated and the `submitBlock` function of the smart contract with the Merkle hash and block number is called
02:10 - the demo script finished working, which sent 1 million transactions in 32 seconds
02:33 - the nodes began to receive information that block # 8 was added to the root chain, and began to perform 240k transactions
02:40 - 240k transactions were deleted from the pool, which are already in block # 8
02:56 - submit node took the remaining 760k transactions from the pool and began to calculate the Merkle hash and sign block # 9
03:20 - all nodes contain 1mln 240k transactions and tokens
03:35 - block # 9 is signed and sent for validation to other nodes
03:41 - a network error has occurred
04:40 - the timeout for waiting for the validation of block # 9 has stopped
04:54 - submit node took the remaining 760k transactions from the pool and began to calculate the Merkle hash and sign block # 9
05:32 - block # 9 is signed and sent for validation to other nodes
05:53 - block # 9 is validated and sent to the root chain
06:17 - nodes began to receive information that block # 9 was added to the root chain and began to perform 760k transactions
06:47 - the pool cleared of transactions that are in block # 9
09:06 - all nodes contain 2 million transactions and tokens
Test 2
There is a limit of 350k per block. As a result, we have 3 blocks.
Initial state: last block # 9; 2 million transactions and tokens stored in the database
00:00 - transaction generation script is already running
00:44 - 1 million transactions were created and sending to the node began
00:56 - the submit node took 320k transactions from the pool and forms block # 10. We also see that 320k transactions are added to the pool in 10 seconds
01:12 - block # 10 is signed and sent to other nodes for validation
01:18 - the demo script finished working, which sent 1 million transactions in 34 seconds
01:20 - block # 10 is validated and sent to the root chain
01:51 - all nodes received information from the root chain that block # 10 has been added, and they begin to apply 320k transactions
02:01 - the pool cleared for 320k transactions that were added to block # 10
02:15 - submit node took 350k transactions from the pool and forms block # 11
02:34 - block # 11 is signed and sent to other nodes for validation
02:51 - block # 11 is validated and sent to the root chain
02:55 - the last node executed transactions from block # 10
10:59 - a very long time in the root chain a transaction was carried out with the submission of block # 9, but it completed and all the nodes received information about this and started to execute 350k transactions
11:05 - the pool was cleared for 320k transactions that were added to block # 11
12:10 - all nodes contain 1 million 670k transactions and tokens
12:17 - submit node took 330k transactions from the pool and forms block # 12
12:32 - block # 12 is signed and sent to other nodes for validation
12:39 - block # 12 is validated and sent to the root chain
13:44 - all nodes received information from the root chain that block # 12 has been added and are starting to apply 330k transactions
14:50 - all nodes contain 2 million transactions and tokens
Test 3
In the first and second servers, one validating node was replaced with a submit node.
Initial state: last block # 84; 0 transactions and tokens are stored in the database
00:00 - 3 scripts are launched that generate and send 1 million transactions
01:38 - 1 million transactions were created and sending to submit node # 3 began
01:50 - submit node # 3 took 330k transactions from the pool and forms block # 85 (f21). We also see that 350k transactions are added to the pool in 10 seconds
01:53 - 1 million transactions were created and sending to submit node # 1 began
01:50 - submit node # 3 took 330k transactions from the pool and forms block # 85 (f21). We also see that 350k transactions are added to the pool in 10 seconds
02:01 - submit node # 1 took 250k transactions from the pool and forms block # 85 (65e)
02:06 - block # 85 (f21) is signed and sent to other nodes for validation
02:08 - the demo script of server # 3 has finished working, which sent 1 million transactions in 30 seconds
02:14 - block # 85 (f21) is validated and sent to the root chain
02:19 - block # 85 (65e) is signed and sent to other nodes for validation
02:22 - 1 million transactions were created and sending to submit node # 2 began
02:27 - block # 85 (65e) is validated and sent to the root chain
02:29 - submit node # 2 took from the pool 111855 transactions and forms block # 85 (256).
02:36 - block # 85 (256) is signed and sent to other nodes for validation
02:36 - the demo script of server # 1 finished working, which sent 1mln transactions in 42.5 seconds
02:38 - block # 85 (256) is validated and sent to the root chain
03:08 - the server script script # 2 finished working, which sent 1 million transactions in 47 seconds
03:38 - all nodes received information from the root chain that the blocks # 85 (f21), # 86 (65e), # 87 (256) were added and begin to apply 330k, 250k, 111855 transactions
03:49 - the pool was cleared at 330k, 250k, 111855 transactions that were added to blocks # 85 (f21), # 86 (65e), # 87 (256)
03:59 - submit node # 1 took from the pool 888145 transactions and forms block # 88 (214), submit node # 2 took from the pool 750k transactions and forms block # 88 (50a), submit node # 3 took from the pool 670k transactions and forms block # 88 (d3b)
04:44 - block # 88 (d3b) is signed and sent to other nodes for validation
04:58 - block # 88 (214) is signed and sent to other nodes for validation
05:11 - block # 88 (50a) is signed and sent to other nodes for validation
05:11 - block # 85 (d3b) is validated and sent to the root chain
05:36 - block # 85 (214) is validated and sent to the root chain
05:43 - all nodes received information from the root chain that blocks # 88 (d3b), # 89 (214) were added and begin to apply 670k, 750k transactions
06:50 - due to a disconnection, block # 85 (50a) was not validated
06:55 - submit node # 2 took 888145 transactions from the pool and forms block # 90 (50a)
08:14 - block # 90 (50a) is signed and sent to other nodes for validation
09:04 - block # 90 (50a) is validated and sent to the root chain
11:23 - all nodes received information from the root chain that block # 90 (50a) was added, and 888145 transactions began to be applied. At the same time, server # 3 has long applied transactions from blocks # 88 (d3b), # 89 (214)
12:11 - all pools are empty
13:41 - all server nodes # 3 contain 3 million transactions and tokens
14:35 - all server nodes # 1 contain 3 million transactions and tokens
19:24 - all server nodes # 2 contain 3 million transactions and tokens
Obstacles
During the development of Plasma Cash, we encountered the following problems, which we gradually solved and are solving:
1. The conflict of interaction of various functions of the system. For example, the function of adding transactions to the pool blocked the submission and validation of blocks, and vice versa, which led to a drawdown in speed.
2. It was not immediately clear how to send a huge number of transactions and at the same time minimize the cost of data transfer.
3. It was not clear how and where to store data in order to achieve high results.
4. It was not clear how to organize a network between nodes, since the block size with 1 million transactions takes about 100 MB.
5. Working in single-threaded mode breaks the connection between nodes when long calculations occur (for example, building a Merkle tree and calculating its hash).
How did we deal with all this?
The first version of the Plasma Cash node was a kind of combiner that could do everything at the same time: accept transactions, submit and validate blocks, provide an API for accessing data. Since NodeJS was initially single-threaded, the heavy Merkle tree calculation function blocked the add transaction function. We saw two options for solving this problem:
1. Run several NodeJS processes, each of which performs certain functions.
2. Use worker_threads and put the execution of the code into threads.
As a result, we took advantage of both options at the same time: logically divided one node into 3 parts, which can work separately, but at the same time synchronously
1. Submit a node that accepts transactions to the pool and creates blocks.
2. Validating node that verifies the validity of nodes.
3. Node API - provides an API for accessing data.
At the same time, you can connect to each node through a unix socket using cli.
Heavy operations, such as the calculation of the Merkle tree, we carried out in a separate stream.
Thus, we achieved the normal operation of all Plasma Cash functions simultaneously and without failures.
As soon as the system functionally worked, we began to test the speed and, unfortunately, got unsatisfactory results: 5,000 transactions per second and up to 50,000 transactions in a block. I had to find out what was implemented incorrectly.
To begin with, we started testing the mechanism of communication with Plasma Cash in order to find out the peak capability of the system. Earlier we wrote that the Plasma Cash node provides a unix socket interface. It was originally textual. json objects were sent using `JSON.parse ()` and `JSON.stringify ()`.
```json { "action": "sendTransaction", "payload":{ "prevHash": "0x8a88cc4217745fd0b4eb161f6923235da10593be66b841d47da86b9cd95d93e0", "prevBlock": 41, "tokenId": "57570139642005649136210751546585740989890521125187435281313126554130572876445", "newOwner": "0x200eabe5b26e547446ae5821622892291632d4f4", "type": "pay", "data": "", "signature": "0xd1107d0c6df15e01e168e631a386363c72206cb75b233f8f3cf883134854967e1cd9b3306cc5c0ce58f0a7397ae9b2487501b56695fe3a3c90ec0f61c7ea4a721c" } } ```
We measured the transfer speed of such objects and received ~ 130k per second. We tried to replace the standard functions with json, but the performance did not improve. There must be a V8 engine well optimized for these operations.
Work with transactions, tokens, blocks was carried out through classes. When creating such classes, performance slipped by 2 times, which indicates: OOP is not suitable for us. I had to rewrite everything on a purely functional approach.
Write to the database
Initially, Redis was chosen for data storage as one of the most productive solutions that satisfies our requirement: key-value storage, work with hash tables, and many. We launched redis-benchmark and got ~ 80k operations per second in 1 pipelining mode.
For high performance, we tuned Redis more finely:
- Established a unix socket connection.
- Disable state saving to disk (for reliability, you can configure the replica and already save to disk in a separate Redis).
In Redis, a pool is a hash table, since we need the ability to receive all transactions in one request and delete transactions one by one. We tried to use a regular list, but it works more slowly when unloading the entire list.
Using the standard NodeJS library, Redis libraries achieved 18k transactions per second performance. Speed dropped 9 times.
Since the benchmark showed us the possibilities clearly 5 times more, they began to optimize. We changed the library to ioredis and got a performance of 25k per second. We added transactions one by one using the `hset` command. Thus, we generated a lot of queries in Redis. There was an idea to merge transactions into bundles and send them with one hmset command. The result is 32k per second.
For several reasons, which will be described below, we work with data using `Buffer` and, as it turned out, if you translate it into text (` buffer.toString ('hex') `) before writing, you can get additional performance. Thus, the speed was increased to 35k per second. At the moment, we decided to suspend further optimization.
We had to switch to a binary protocol because:
1. The system often calculates hashes, signatures, etc., and for this it needs data in `Buffer.
2. When transferring between services, binary data weighs less than text. For example, when sending a block with 1 million transactions, the data in the text can occupy more than 300 megabytes.
3. Continuous data conversion affects performance.
Therefore, we took as our basis our own binary protocol for storing and transmitting data, developed on the basis of the remarkable library `binary-data`.
As a result, we have the following data structures:
- Transaction
```json { prevHash: BD.types.buffer(20), prevBlock: BD.types.uint24le, tokenId: BD.types.string(null), type: BD.types.uint8, newOwner: BD.types.buffer(20), dataLength: BD.types.uint24le, data: BD.types.buffer(({current}) => current.dataLength), signature: BD.types.buffer(65), hash: BD.types.buffer(32), blockNumber: BD.types.uint24le, timestamp: BD.types.uint48le, } ```
- Token
```json { id: BD.types.string(null), owner: BD.types.buffer(20), block: BD.types.uint24le, amount: BD.types.string(null), } ```
- Block
```json { number: BD.types.uint24le, merkleRootHash: BD.types.buffer(32), signature: BD.types.buffer(65), countTx: BD.types.uint24le, transactions: BD.types.array(Transaction.Protocol, ({current}) => current.countTx), timestamp: BD.types.uint48le, } ```
By the usual commands `BD.encode (block, Protocol) .slice ();` and `BD.decode (buffer, Protocol)` we convert the data to `Buffer` to save it to Redis or transfer it to another node and retrieve the data back.
We also have 2 binary protocols for transferring data between services:
- Protocol for interacting with Plasma Node via unix socket
```json { type: BD.types.uint8, messageId: BD.types.uint24le, error: BD.types.uint8, length: BD.types.uint24le, payload: BD.types.buffer(({node}) => node.length) } ```
Where:
- `type` - action to be performed, for example, 1 - sendTransaction, 2 - getTransaction;
- `payload` - data to be transferred to the corresponding function;
- `messageId` - message id so that the response can be identified.
- Protocol of interaction between nodes
```json { code: BD.types.uint8, versionProtocol: BD.types.uint24le, seq: BD.types.uint8, countChunk: BD.types.uint24le, chunkNumber: BD.types.uint24le, length: BD.types.uint24le, payload: BD.types.buffer(({node}) => node.length) } ```
Where:
- `code` - message code, for example 6 - PREPARE_NEW_BLOCK, 7 - BLOCK_VALID, 8 - BLOCK_COMMIT;
- `versionProtocol` - protocol version, since nodes with different versions can be raised on the network and they can work in different ways;
- `seq` - message identifier;
- `countChunk` and` chunkNumber` are needed to split large messages;
- `length` and` payload` the length and the data itself.
Since we typed the data beforehand, the final system is much faster than the `rlp` library from Ethereum. Unfortunately, we have not yet been able to refuse it, since it is necessary to finalize the smart contract, which we plan to do in the future.
If we managed to achieve a speed of 35,000 transactions per second, we also need to process them in optimal time. Since the approximate formation time of the block takes 30 seconds, we need to include 1,000,000 transactions in the block, which means sending more than 100 mb of data.
Initially, we used the `ethereumjs-devp2p` library to communicate nodes, but it could not cope with so much data. As a result, we used the `ws` library and set up binary data forwarding via websocket. Of course, we also encountered problems when sending large data packets, but we divided them into chunks and now there are no such problems.
Also, the formation of the Merkle tree and the calculation of the hash of 1,000,000 transactions requires about 10 seconds of continuous calculation. During this time, the connection to all nodes manages to break. It was decided to transfer this calculation to a separate thread.
Findings:
In fact, our findings are not new, but for some reason, many experts forget about them during development.
- Using Functional Programming instead of Object-Oriented Programming increases performance.
- A monolith is worse than a service architecture for a production system on NodeJS.
- Using `worker_threads` for heavy computing improves the responsiveness of the system, especially when working with i / o operations.
- unix socket is more stable and faster than http requests.
- If you need to quickly transfer large data over the network, it is better to use websockets and send binary data, broken into chunks, which can be forwarded if they do not reach, and then merged into one message.
We invite you to visit the GitHub project: https://github.com/opporty-com/Plasma-Cash/tree/new-version
The article was co-written by Alexander Nashivan , senior developer of Clever Solution Inc.
All Articles