Technical problems, or what to think if you decide to hold your own ICO
The interest we have actively fueled for Polybius' ICO turned into a resounding success - and this time almost in the literal sense of the word. Stunned by the attention of 14,000 people who came to buy our tokens in the first minutes after the start of sales, the site of Polybius collapsed. And that was just the beginning. As promised, we tell what happened on May 31 and what we could do better to avoid what happened.
Of course, at the start of ICO, we were ready for both DDoS attacks and high server load, however, we admit, we simply underestimated both the capabilities of malicious hackers and the degree of interest in the project, and the likelihood of emergency situations.
Ambisafe provided the technical platform (investor’s office, the entire infrastructure) for the Polybius ICO. Starting with the development of blockchain products, the company eventually expanded its interests towards providing firms with a set of services and tools necessary for holding an ICO. Having chosen an experienced contractor, we fully relied on their knowledge and skills, without having supervised once again those things that we could (and should have!) Controlled. And it was our first mistake.
Good server power is a crucial detail if you don’t want the site to collapse at the start of sales. Predicting the possible load on the servers during the first hours of an ICO, the contractor provided us with requirements for their capacity, but even then we assumed that these indicators might not be enough and doubled the performance. To share the load between the servers, we also located the user's office and the Polybius site in different places, but did not separate the database and the main cabinet files. And it was a mistake number two.
The database suffered every time when too many requests came to the investor’s office (and vice versa), and the server crashed, periodically becoming unavailable. When the problem became apparent, we increased the capacity of the servers by almost 10 times compared with the original characteristics. However, it took time to change the configuration of the cabinet and re-upload all the files to the new servers.
The rush around the start of the Polybius ICO was so strong that as soon as the countdown counter on the polybius.io website reached the zero point, crowds of users rushed to buy tokens, and some of our users even managed to make a false start. By itself, the excitement of real users is not a big problem. The problem was increased attention to the ICO of Polybius from the side of the bots - a DDoS attack began. In about 6 hours, about 30 million inquiries from the personal account of the investor came from the ddosers, which simply choked them. The situation was aggravated by the fact that the first watch was doubly difficult to fight off malicious traffic - it was not always clear, this is a real person sitting and pressing “Refresh” in the hope of seeing a personal account, or it is a bot that does the same.
Since there were a lot of people willing to participate at the start, they aggressively made their way to the site through DDoS and continued to buy tokens when the site became easier. As a result, the system has accumulated a huge number of transactions that were not executed on time and stuck inside it. And here we smoothly move on to another problem that, for obvious reasons, worried the crowdfunding participants the most.
Since the system in which the transactions took place is large and multi-module, each operation had to go through a clear route. Because of the DDoS attacks in certain places of the system, the modules began to slow down, so simultaneously with filtering out malicious traffic, it was necessary to almost manually disassemble the transactions stuck in the system. But even after pushing through stuck operations, we could not ensure timely receipt of purchased tokens due to obstruction in the Ethereum network. It was an emergency situation, which gave rise to panic and a flurry of comments like “ I was withdrawn money, but the tokens were not credited. It has been half an hour / hour / 5 hours ... ”Now that tokens have long been accrued to all affected investors, we can only apologize once again for the situation and explain what caused it.
At the time of the start of the ICO Polybius on the Ethereum blockchain of the network, another big ICO (BAT - BasicAttentionToken ) took place, and competition between investors led to network overload. To understand those of our readers who do not know so much about Ethereum - the network works in such a way that in order to send a transaction to it, you must pay a commission. And the higher the commission, the faster the transaction enters the network. Against the background of the second ICO, whose members started to set up huge commissions, Polybius’s transactions with the optimal commission in terms of cost and time for confirmation stopped being confirmed and queued in the Ethereum network itself, this time stuck somewhere on the border between our system blockchain.
In order to conduct in Ethereum, not a single, but a batch of transactions, as was the case in our case, they need to be queued up by assigning (conditionally) a serial number to each transaction. Sequence numbers must be in absolute increment - that is, each subsequent transaction must have a number one greater than the previous one. So, we cannot send transaction number 10 to the network until the first nine are processed. In order not to load the network even more and not to increase the number of stuck transactions, we were forced to increase the commission for all new orders - new transactions were confirmed in the normal mode and in a more acceptable time frame. The very first transactions (we are talking about a huge number of orders - more than a thousand) had to be left "as is", so it took several days before the first investors could receive the tokens belonging to them.
This is an abnormal system behavior and a situation that our contractors had to decide for the first time, so it’s difficult to judge whether the optimal tactics were chosen, or we could have done more.
The most important, if not unique, understanding is that when the problem has already manifested itself, all you can do is start to solve it as quickly as possible. The first thing we did was to increase the server’s characteristics, take more powerful hardware, split the database across different servers, set up a firewall, filter out DDoS attacks. And all these processes took about a day. Additional time was needed to establish the transaction process through Ethereum.
Now that we already have ICO experience, we can safely say that it is impossible to be too prudent in this matter. It is better to be safe once again, so if you suddenly decide to hold your own ICO, please think immediately that:
And, most importantly, you will never be able to foresee all possible problems, so just be prepared for any surprises. You can spend a lot of time counting mistakes made inadvertently, but learn from mistakes, and we are glad that they did not become a serious obstacle to the successful completion of our campaign.
Of course, at the start of ICO, we were ready for both DDoS attacks and high server load, however, we admit, we simply underestimated both the capabilities of malicious hackers and the degree of interest in the project, and the likelihood of emergency situations.
Ambisafe provided the technical platform (investor’s office, the entire infrastructure) for the Polybius ICO. Starting with the development of blockchain products, the company eventually expanded its interests towards providing firms with a set of services and tools necessary for holding an ICO. Having chosen an experienced contractor, we fully relied on their knowledge and skills, without having supervised once again those things that we could (and should have!) Controlled. And it was our first mistake.
Iron too weak
Good server power is a crucial detail if you don’t want the site to collapse at the start of sales. Predicting the possible load on the servers during the first hours of an ICO, the contractor provided us with requirements for their capacity, but even then we assumed that these indicators might not be enough and doubled the performance. To share the load between the servers, we also located the user's office and the Polybius site in different places, but did not separate the database and the main cabinet files. And it was a mistake number two.
The database suffered every time when too many requests came to the investor’s office (and vice versa), and the server crashed, periodically becoming unavailable. When the problem became apparent, we increased the capacity of the servers by almost 10 times compared with the original characteristics. However, it took time to change the configuration of the cabinet and re-upload all the files to the new servers.
The rush around the start of the Polybius ICO was so strong that as soon as the countdown counter on the polybius.io website reached the zero point, crowds of users rushed to buy tokens, and some of our users even managed to make a false start. By itself, the excitement of real users is not a big problem. The problem was increased attention to the ICO of Polybius from the side of the bots - a DDoS attack began. In about 6 hours, about 30 million inquiries from the personal account of the investor came from the ddosers, which simply choked them. The situation was aggravated by the fact that the first watch was doubly difficult to fight off malicious traffic - it was not always clear, this is a real person sitting and pressing “Refresh” in the hope of seeing a personal account, or it is a bot that does the same.
Since there were a lot of people willing to participate at the start, they aggressively made their way to the site through DDoS and continued to buy tokens when the site became easier. As a result, the system has accumulated a huge number of transactions that were not executed on time and stuck inside it. And here we smoothly move on to another problem that, for obvious reasons, worried the crowdfunding participants the most.
Network overload
Since the system in which the transactions took place is large and multi-module, each operation had to go through a clear route. Because of the DDoS attacks in certain places of the system, the modules began to slow down, so simultaneously with filtering out malicious traffic, it was necessary to almost manually disassemble the transactions stuck in the system. But even after pushing through stuck operations, we could not ensure timely receipt of purchased tokens due to obstruction in the Ethereum network. It was an emergency situation, which gave rise to panic and a flurry of comments like “ I was withdrawn money, but the tokens were not credited. It has been half an hour / hour / 5 hours ... ”Now that tokens have long been accrued to all affected investors, we can only apologize once again for the situation and explain what caused it.
At the time of the start of the ICO Polybius on the Ethereum blockchain of the network, another big ICO (BAT - BasicAttentionToken ) took place, and competition between investors led to network overload. To understand those of our readers who do not know so much about Ethereum - the network works in such a way that in order to send a transaction to it, you must pay a commission. And the higher the commission, the faster the transaction enters the network. Against the background of the second ICO, whose members started to set up huge commissions, Polybius’s transactions with the optimal commission in terms of cost and time for confirmation stopped being confirmed and queued in the Ethereum network itself, this time stuck somewhere on the border between our system blockchain.
In order to conduct in Ethereum, not a single, but a batch of transactions, as was the case in our case, they need to be queued up by assigning (conditionally) a serial number to each transaction. Sequence numbers must be in absolute increment - that is, each subsequent transaction must have a number one greater than the previous one. So, we cannot send transaction number 10 to the network until the first nine are processed. In order not to load the network even more and not to increase the number of stuck transactions, we were forced to increase the commission for all new orders - new transactions were confirmed in the normal mode and in a more acceptable time frame. The very first transactions (we are talking about a huge number of orders - more than a thousand) had to be left "as is", so it took several days before the first investors could receive the tokens belonging to them.
This is an abnormal system behavior and a situation that our contractors had to decide for the first time, so it’s difficult to judge whether the optimal tactics were chosen, or we could have done more.
What we have learned
The most important, if not unique, understanding is that when the problem has already manifested itself, all you can do is start to solve it as quickly as possible. The first thing we did was to increase the server’s characteristics, take more powerful hardware, split the database across different servers, set up a firewall, filter out DDoS attacks. And all these processes took about a day. Additional time was needed to establish the transaction process through Ethereum.
Now that we already have ICO experience, we can safely say that it is impossible to be too prudent in this matter. It is better to be safe once again, so if you suddenly decide to hold your own ICO, please think immediately that:
- the load in the first hours can be just going off-scale and exceed even the wildest expectations;
- if your project is heard, then attention from ddosers is guaranteed to you, so take care of protection against possible attacks;
- if 10 more ICOs will pass along with yours, it is better to spend time thinking about what tactics to choose - whether to wait several days, when transactions one way or another, but pass, or to invest additional resources and force the execution of orders through the network;
- It is worth paying more attention to timely informing investors about the problems that have arisen - this will definitely help reduce the number of identical requests for technical support and free up its resources for solving other pressing problems.
And, most importantly, you will never be able to foresee all possible problems, so just be prepared for any surprises. You can spend a lot of time counting mistakes made inadvertently, but learn from mistakes, and we are glad that they did not become a serious obstacle to the successful completion of our campaign.
All Articles