On the eve of Black Friday, the most popular and largest (but far from the only) global sale, it is worth taking care of the accessibility of your sites. How? Specialists of IT-GRAD will give advice under the cut.
/ Original Image - Shutterstock
The main reason for the "fall" of sites during the sales period is a sharp increase in the amount of traffic and not properly prepared infrastructure. It should be remembered that in parallel with the natural influx of visitors, a DDoS attack is possible.
Therefore, the surest way to survive, for example, Black Friday and not to fall, is to lay the opportunity for scaling for the peak period. The architecture of the solution must be built in such a way that adding resources can solve this problem. The same applies to the code of the application itself: it must take into account the possibility of scaling. However, we will give some practical advice at the very end of the article.
Of course, it is impossible to create the perfect step-by-step instruction "To never fall, you need only ...", which will suit everyone and everyone. Nevertheless, in the midst of autumn, when the hottest sales start all over the world one after another, it will not be superfluous to recall some rules of preparation, but at the same time to study the invaluable experience of our colleagues.
In no case do we seek to discredit any of the companies whose cases will be considered in the article. The sole purpose of mentioning the incidents below is to illustrate the risks, protection methods, and ways to mitigate the consequences of a denial of service. All calculations of estimated losses were carried out by us only approximately and cannot reflect the real picture in companies.
Huawei
Date of incident: April 4, 2018.
What happened: It would seem that China is one of the most densely populated countries in the world. And the Chinese vendor, which launched an action of unprecedented generosity (smartphones and tablets for only 1000 rubles), could well predict a serious load. However, the company’s office underestimated the enthusiasm of our hi-tech lovers, as a result of which the website of the Russian division of Huawei stopped opening.
The failure was recorded literally immediately after the start of the sale, at about 11:00 Moscow time. At the same time, the main "promotional" page opened without problems, but when trying to add the product to the cart, an HTTP Status 404 - / CAS / remoteLogin error occurred. Company representatives refrained from commenting, citing, again, an unplanned increase in traffic during the campaign.
Expert commentary: Huawei started everything quite peacefully. In fact, the campaign spoke of only 10 devices for 1000 rubles, but people didn’t understand something, and it started. It is likely that the true cause of the fall was not enough quality marketing - when developing advertising materials, it was worthwhile to clearly indicate the limits.
Fortunately, the company avoided reputational damage . Most users were sympathetic to the technical problems on the site, and the brand's positive reputation made it possible to hush up the situation.
Victory
Incident Date: August 12, 2019
What happened: The website of the Pobeda low-cost airline (or rather, the reservation service) stopped working correctly during the sale of tickets at 499 rubles.
According to the representatives of "Victory", the site was subjected to "a powerful DDoS attack from Chinese servers." They said that there were no interruptions in work with the influx of visitors and they started earlier than the action started. When trying to find tickets, the site gave error No. 502 and a message about the unavailability of the service.
For Victory, this is not the first case of a fall during a sale - the same thing happened before.
As a result: Pobeda planned to sell 200,000 tickets at the lowest rates - from 499 to 1999 rubles. According to the company, 160 thousand were added to the site, but out of the planned number it was still possible to sell about 52,203 tickets. Accordingly, 107 797 tickets remained, the average price of which was ~ 1249 rubles. Based on this, we can make a rough estimate and calculate that the “Victory” could maximize another 134 million rubles.
However, the very fact that the sale of tickets caused such a stir, and Pobeda received a huge number of references in the media and social networks suggests that tickets of 499 rubles can make users forget about the fall and other dubious moments.
Singles Day on AliExpress
Date of incident: November 11, 2015
What happened: The total sale, timed to coincide with the "Bachelor's Day", popular in China, was held in Russia for the first time in 2015. However, users and partners of the popular trading platform encountered problems when paying for goods. For several hours, issued and paid orders remained in users' baskets with the status “Waiting for payment”, and money was deducted from the cards twice.
As a result: Nevertheless, representatives of AliExpress stated that there were no failures on their side and the culprits of the current situation were some large Russian and European banks that could not cope with the sharply increased number of requests for payment.
Talk about reputation. The crash affected many users from Russia and Europe. Moreover, it entailed material costs for some of these users. Nevertheless, there is a positive moment for retail: despite the drop, this proves that people were so passionate about the action that even banks could not “digest” all payments.
BlackFriday2013.ru
Date of incident: December 6, 2013
What happened: The Black Friday sale site refused literally right after the sale started. Ozon, Enter, White Wind, L'Etoile and other partners of the action, whose banners were placed on the organizer's page, also took part in the festive fall.
As a matter of urgency, technical work was carried out on the site, during which the xml feeds were incorrectly uploaded, and buyers received such discount offers:
/ So the result of incorrectly filled xml feeds looked
As a result: According to the organizers, due to the fact that Black Friday was held in our country for the first time in 2013, they could not correctly assess the estimated load. It took about 11 hours to restore performance, the action was extended until Saturday evening. Despite the positive feedback from representatives of Ozon, Bukvoed and other participating companies, some of the partners in 2013 refused to participate in future promotions. It is likely that a significant malfunction in the work of a dedicated aggregator site and, as a result, a simple one in sales is one of the reasons. In this case, it is worth noting that shifting part of the load to the sales aggregator is a way to reduce risks for retailers, however, in this case, the aggregator assumes considerable responsibility for ensuring the action.
Belavia
Date of incident: March 5, 2016
What happened: on the eve of the accident, the Belarusian airline Belavia, in honor of the twentieth anniversary, announced the sale of tickets for only 20 euros. It was estimated that up to 114,000 tickets would be sold at a special rate throughout the day on March 5th.
Nevertheless, it turned out that the company's servers were not ready for a record influx of visitors. Just an hour after the start of the action, users inundated social networks with messages about malfunctions on the site.
/ Screenshot of publication from social networks
Expert commentary: judging by statements on social networks, Belavia at that time used physical infrastructure. In such a situation, any extension takes a certain time, even in emergency mode. Using cloud resources would help make scaling smoother.
As a result: Belavia planned to sell 114,000 tickets as much as possible, according to the report, 47,681 tickets were sold. However, this was facilitated by the extension of the action for one more day - on March 5, the company sold only 10,000 tickets.
/ Screenshot of publication from social networks
/ Photo from the Belavia presentation summarizing the results of 2016, blog.vp.by
Reputation: Belavia may have found the most elegant and honest way to restore a tarnished reputation. The action was extended for another day, and on March 6, the sale of tickets for the promotion tariff resumed, this time without a glitch.
Draw conclusions
In conclusion, we present a checklist, compliance with which before the expected load peaks will help your site survive. The list of recommendations is divided into 2 parts: the first is devoted to training on the part of management and marketing, the second to engineers and operation specialists.
Management and Marketing
- Very often, stocks and sales are planned for at least six months, and based on past years, you can make an approximate forecast for load growth.
- Form the most honest and understandable offers for customers to avoid unnecessary excitement. A vaguely worded action, multiplied by a slowing down site, can cause serious reputation and financial losses.
- Think about the possibility of using affiliate platforms to more evenly distribute traffic during the campaign.
- If you still have the logs of past years, do not be too lazy to pick them up and study the behavior of site users.
- Coordinate the action plan and other activities that may cause traffic growth with the IT department.
IT
- IT infrastructure and application architecture must be scalable.
- Make sure the infrastructure accepts the potential load without any problems. Ideally, it is worthwhile to conduct load testing in advance and modify it if problems are identified.
- Notify your provider of upcoming peak if using cloud resources. Check the possibility of "hot" adding computing resources: adding resource pools, adding virtual machine resources.
- Separately, consider scalability at the application level.
- Highlight typical user paths on your site and create several scenarios of real load emulation based on them.
- If possible, deploy a test bench and conduct load testing according to previously prepared scenarios.
- Deploy active-active support and expand your active distributed infrastructure.
- Work on network connectivity across the entire chain of active network devices and channels from the application server to the edge of the infrastructure.
If you have any questions, we will be glad to talk in the comments.