Clever Geek Handbook

5 GHz Iron: A Story on How to Build a Superfast VDS





If our good old tariff "Fast and the Furious" was a man, now he would nervously smoke aside and recall the best years. But how well it all began. Then, in 2018, we were the first to connect NVMe and modify the tariff with flexible scaling of resources, which has lived with us since 2016. Then he was the fastest of all our tariffs. But times are changing, and despite the fact that the Gold 5115 is still considered a rather nimble processor, solutions have already appeared and frolic.



In general, my story is about how the Fast and the Furious had to reluctantly leave the top step of the podium and take third place in speed after two new tariffs - Bitrix.Turbo and CPU.Turbo, which we launched during this year.



I can promise that there will be a lot of details about hardware, tests of processors and (well, necessary) virtual machines and some internal kitchen. So let's get started.



This is a survival race - now it will be fun!



In February 2019, we launched a new Bitrix tariff: Turbo: servers with Intel Core i9-9900K processors with a clock frequency of 5 GHz in Turbo Boost mode. We drove performance tests - then I wrote my first article , and the guys from the neighboring department sawed the recipe to get the most out of Bitrix . i9 met our expectations, the launch took place. Thank you all, diverge ...



In fact, Bitrix Turbo turned out to be very popular among our customers. Until now, we only have time to add servers to the cluster. Therefore, we made conclusions and decided not to stop there, but to continue our research in the field of high-frequency processors.



Since Bitrix.Turbo was conceived (and it turned out!) By a very specialized tariff, and Fast and the Furious customers appreciated more for flexibility, it was decided to supplement the line with a high-speed tariff for all those who need "even cooler, even faster."



Since the i9-9900K presented us with a couple of “surprises,” we immediately decided that we would be launching on other processors. Yes, that’s how it was, already at the stage of designing a hardware solution for Bitrix. Turbo, we had a “fascinating and rich experience”. Well, or he is us. Since this is the case, I will deviate a little from the main narrative and literally briefly tell about this experience.



A standard server for the Bitrix tariff. The turbo is assembled on a single-unit platform: a motherboard based on the Intel Q370 chipset, an Intel i9-9900K processor, 64 GB RAM, two 2 TB NVMe drives in RAID1.



Implementing such a solution was extremely fun and interesting. The fact is that according to Intel, the TDP of this processor is 95 watts. Indeed, if you use motherboards, for example, with the H310 chipset, the processor at peak load does not go beyond these limits: it quietly accelerates one core to 5 GHz and the rest in the range from 4.7 to 4.8 GHz. But if the platform uses a chipset of a class higher, for example, Q370 and improved processor power, then the processor accelerates to a larger number of cores up to 5 GHz: from 2 to 5, and TDP increases to almost 200 watts.



It is not an easy task to cool such a “stove” in a single unit design. The processor began to throttle and lowered the frequency on all cores to 4.4-4.5 GHz. It was extremely difficult to control such a situation. Honestly, for the first time I came across such a strong discrepancy between the specifications and the actual implementation of such a large and serious manufacturer as the Blues.



Due to the fact that at that time it was the only available processor with such high nominal frequencies, we had to work hard before allowing ourselves to use it for a new tariff, and resort to rather non-trivial solutions: use special cases with powerful air cooling and organize no less cool cooling racks. All this together allows you to make more cores work at maximum at a frequency of 5 GHz without overheating and throttling, but it increases power consumption and noise, wild noise.



Unfortunately, at that moment in time, we could not introduce liquid cooling: on the one hand, there was no suitable solution in the right amount to start, on the other, there were a lot of questions about the reliability of existing ones - we really did not want to drown the servers.



Now back to the main story.



Rake Dancing







As you know, processor manufacturers test their solutions on the desktop market and, in case of a positive result, implement them in the server segment. We could only wait for the appearance of the server analog of Intel i9-9900K. And almost a year later, such an analog appeared - it was Intel Xeon E-2288G. Intel took into account the shortcomings of its predecessor: the processor has become tightly limited in the heat packet (95 W). As with any server processor from Intel, ECC-enabled memory has appeared, and the maximum supported capacity has grown to 128 GB.



We figured out the processor, it's time to deal with the server platform. Finding the right one became a serious problem for us, since we were looking for one that would suit us according to the tactical and technical characteristics, had an adequate cost and supported all the possibilities of such a fast stone.



Since the processor is new, the choice of platforms has been very, very limited. After a thorough and meticulous study, we settled on three options: the first fell off during the first tests, - the manufacturer released a crude release of the platform firmware with support for E-2288G, with which the whole solution worked extremely unstable, and it was only possible to wait for a new release and platform stabilization of the year. The second did not allow to organize full cooling of the power circuits, which is very important for this processor. The new processor was too tough for them. But the third one came up in all respects. Not only met, but even exceeded our expectations. I can’t disclose models - a trade secret.



We also decided to abandon extreme air cooling in favor of liquid (hereinafter “water”), to which we are gradually planning to transfer servers for the Bitrix.Turbo project. Progress does not stand still, and it is wonderful.



As a result, the transition to water cooling allowed us to increase the performance of both the Intel i9-9900K and Intel Xeon E-2288G, without passing the temperature threshold of 65 degrees in the maximum continuous load in all flows. Having spent a lot of time working out all the possibilities, as well as all possible and even impossible emergency situations (I was very skeptical of “water”), having driven out lengthy tests at maximum loads, even I was forced to agree that this solution deserves to live long and happily. Therefore, get acquainted - the new CPU rate. Turbo.



The performance characteristics of hardware and software for the new tariff



Iron Single-unit single-processor platform, Intel Xeon E-2288G processor with water cooling system, 64 GB ECC RAM, disk subsystem on two Intel M.2 NVMe drives in RAID1.



Soft Since we use ISPsystem VMmanager 5 panels, the choice of OS is not very rich, therefore Linux CentOS 7 OS with Oracle Unbreakable Enterprise Kernel Release 5 (UEK5) 4.14, qemu 2.12, libvirt 4.5.



Testing



This time the testing was carried out in two runs.



First, we compared the processors: i9-9900K on “air”, i9-9900K on “water” and E-2288G on “water”. They did not begin to carry out a full set of tests on the latter with air cooling because of the senselessness of the undertaking - it was immediately clear what and how it would be. Also, I did not drive the Intel Gold 5115 processor, which is used on the Fast and the Furious tariff, into the test. These servers come in a dual-processor configuration and in multi-threaded tests, the purity of comparison is lost.



Then we started testing five virtual machines: one on the Fast and the Furious tariff and four on the CPU. Turbo. Testing was carried out with the host processor emulated on an empty parent server (without neighbors) in the “2 core / 4 gig” tariff variations (this is the most popular of the comparable configuration on the Fast and the Furious).







The performance characteristics of the platforms



Bitrix Turbo Tariff





CPU Tariff Turbo





The list of tests that were conducted:



  1. Geekbench
  2. Sysbench
  3. Phoronix test suite


A detailed description of this complex is in my last article . Once I gather my courage, add details and publish separately.





Processor Test Results







According to the Geekbench test in single-threaded mode, the Xeon E-2288G is slightly behind the i9-9900K - the difference is only 0.8%. In multithreaded mode, the i9-9900k is already behind - by 7.63%.



Other test results
























In all tests, with the exception of GeekBench, the i9-9900K on “water” is better than itself on “air”. E-2288G is better than its "progenitor" in all tests, except for the single-threaded GeekBench test and Apache test. I believe that the results in these tests were affected by internal processor optimizations, which allowed us to pack it in a stable heat package. In other single-threaded tests, performance is on the same level with an error - we will not jump above 5 GHz :-)



We are going further.



Virtual Server Testing



We will compare as many as five virtual machines.



First of all, as always, a virtual machine running CentOS Linux 7 with the stock kernel 3.10. Also, due to the fact that the host server uses a custom kernel Unbreakable Enterprise Kernel Release 5 (UEK5) from Oracle, which allows you to increase performance on new processors, let's see how the performance of a virtual machine with CentOS 7 and the UEK5 kernel (4.14) inside will change at the CPU rate. Turbo. If I already took the CentOS 7 test with two different kernels, why not look at the tests passing by virtual machines on other Linux operating systems and other kernel versions? Take the latest versions of Ubuntu LTS and Debian: Ubuntu 18.04 (kernel 4.15) and Debian 10 (kernel 4.19).



So, at the starting line:





Thus, we will compare two tariffs, and two CentOS 7 with different kernels, and 3 versions of popular OS. I have long wanted to stir up such a thing, I did not know which article to put in :-)



The tests are the same, well, I really love them.



Virtual Server Test Results







According to the results of the Geekbench test in single-threaded mode, the first place is in the VM CPU. Turbo with the UEK5 core. At the same time, the difference between Fast and the Furious and CPU. Turbo (3.10) is already 65.02%, and with the UEK5 core - all 67.01%. The difference between CPU. Turbo on Centos 7 and Debian is 0.2% in favor of UEK5.



In multi-threaded mode, CPU is the leader. Turbo Debian. It is ahead of CPU. Turbo (3.10) by 2.19%, and CPU. Turbo (UEK) by 1.13%. At the same time, CPU. Turbo (3.10) is ahead of Fast and Furious by 55.48%. The difference between CPU. Turbo (UEK) and Fast and the Furious reaches 57.13%.



Other test results


























In the single-threaded and multi-threaded Geekbench test, as well as the Sysbench, C-ray, OpenSSL tests, CentOS 7 with both kernels and Debian 10 are at approximately the same level. Ubuntu is on the last place 18.04.



In the compression test: Compress-7zip and Compress-bzip - the first place in Debian 10, the second place is shared by CentOS 7 with both kernels, the third place is in Ubuntu 18.04.



In the encode-mp3 test, the places were distributed as follows: Debian 10, Ubuntu 18.04, CentOS 7 UEK5, CentOS 7 with the kernel 3.10.



In the single-threaded Himeno test, CentOS 7 3.10 and CentOS 7 UEK5 ranked first and second, Debian 10 was the third, and Ubuntu 18.04 was the fourth with a big lag.



In the ffmpeg test, the first place is on CentOS 7 UEK5, the second is on Debian 10, the third place is shared by CentOS 7 3.10 and Ubuntu 18.04.



In the John The Ripper test, Debian 10 took first place, the second with both CentOS 7 with the same result, the third with Ubuntu 18.04.



In the Apache test, CentOS7 UEK5 leads by a wide margin, followed by Debian 10, CentOS 7 3.10, Ubuntu 18.04.



In the GCC test, the difference between VMs on the CPU tariff. Turbo is insignificant.



In all tests, VM has the last place on the Fast and the Furious tariff - there is a difference in frequencies.







Interestingly passed the test of virtual machines. The lag of VM on the Fast and Furious tariff was obvious. The results for VMs with CentOS 7 with different kernels were also obvious: they should have been close or with a slight margin towards a more recent kernel. Only the Apache test was unexpected. But the full revelation was the results of Debian 10 and Ubuntu 18.04 - leading in most tests of Debian 10 and lagging behind in most tests of Ubuntu 18.04.



It’s hard for me to say what exactly influences such a spread of results. Perhaps a more recent kernel and software. Or a set of services launched “out of the box”: tests were conducted on freshly deployed VMs without any settings. If Debian 10 after launch starts a very small number of services that can be counted on the fingers of two hands of an experienced router, then Ubuntu 18.04 did not begin to pack itself into such a narrow framework :-)



In general, for those who decide on their own experience to make sure that the new CPU rate. Turbo is rightly called a high-speed, I recommend taking into account two points. Firstly, the OS - on Debian it will be faster without shamanism, with CentOS 7 it’s also excellent, but if you conjure with the kernel, it’s super. Who is interested in installing UEK5 on CentOS 7 - in the comment. And secondly, it should be remembered that we still tested on empty nodes. And so, under load, the results will be slightly different, but still high. Good to all.



Until October 3, the servers are on the CPU tariff. Turbo can be ordered on the firstvds.ru website with a 25% discount for 1 month using the HABR_TURBO promotional code. The promotional code will burn at midnight on October 4, 2019. You can also take the server for a free three-day test.







PS In all photos our HR Generalist Sonya.



Tested and written for you by Trashwind , senior system administrator of the operations department.


More articles:


All Articles