Light green tour

The main transport problem in St. Petersburg is bridges. At night, they have to run away from the tavern without drinking beer. Well, or pay a taxi twice as much as usual. In the morning - carefully calculate the time so that as soon as the bridge is brought down, you have managed to get to the station with a deft mongoose. We do not consider the option of “spending the night somewhere in the center, next to the station,” as ideologically harmful.

... "Peregrine Falcon", leaving at 5:30, was invented by people who hate owls. No, of course, it’s very convenient to be in Moscow at nine in the morning, and Peter is insanely beautiful during the white nights, but at half past four, when I got in a taxi, I wanted to turn my head by 270 degrees to the lark that this morning train invented.

“All good data centers are alike,” I thought when I heard that the guys from DataLine had invited us to visit.

As it turned out, each data center is good (or bad) a little in its own way, especially when you get bumps, collect all the rakes in the district and learn lessons from your own (and, a little, alien) rich experience. Here we are (as part of the five leading linkmeups and friends and sympathizers who joined them) and decided to check how good (or not so) the data centers OST and NORD are.

Having managed to work a little on the road (there could have been an advertisement for one mobile operator, but no - the coverage was still bad and it remained), we are unloaded at the Leningrad station. Well, at least not under Gazmanov.

We began our tour of the "light green" data centers with OST - the "one" that "burned" not so long ago. Why in quotation marks - after. In the meantime - we stand, we smoke, we are waiting for the late. We photograph the office of RT in the neighborhood. There are many funny stories with this neighborhood - starting with laying fresh asphalt and painting curbs borders before the arrival of the High Personals, and ending with discounts in the surrounding cafes, which the Dalai people did because of the similarity of the badges.

The mouthpiece of totalitarian propaganda

Data center strict (access) mode

The pass, the rules of conduct, the meat grinder at the entrance - almost the same we saw during an excursion to Linxdatacenter . Have you seen a lot of coffee houses in the data centers? We, now, know a couple now. Employees and customers on the badges have "coffee" deposits, which are regularly replenished. so that within a reasonable range of coffee you can treat yourself for free. We, as dear guests, were also treated.

I can’t say anything bad about coffee - it’s usually for a “networked” coffee shop, but the location is involuntarily captivating, and the interior is in the best modern traditions.

The interior of a local coffee shop

In the photos data centers NORD

The tour starts at the office. In addition to the graffiti known on the whole Runet and the white and light green gamut, there are also all sorts of funny things. This is the props left from the performances of the corporate theater (can you imagine the technical director of the network of data centers playing in the play? And we personally know him.).

And the truth

Photos from the performances

The data center (or rather, the four data centers) are located in a complex of industrial buildings built back in the century before last (then the Citroen plant, and in the days of the USSR the Hydrodrive plant), so on the one hand it’s a harsh industrial, metal beams, loft atmosphere and industrial artifacts of different eras, which are still found during various works. On the other hand, the data center pressure zones need to be integrated into existing structures, and periodically compromise with reality. However, I would not say that these compromises affected quality - but the guys probably had to sweat during the design. Still, according to rumors, Metro-2 passes somewhere nearby, but this is not accurate.

In addition to the theater, there is also a museum. Exhibits, natural - archaeological finds

A few words about the fire . Yes, the building (or rather the roof) really burned. The external units of the cooling systems and the cold track were damaged. Somewhere the wiring burned out and the air conditioners were left without electricity. Somewhere freon pipelines exploded. But, in spite of this, the pressure zones withstood tons of water poured onto them, the equipment survived. They were restored under the control of the “headquarters” in the person of the top leadership of the whole world — they raked out all the copper pipes in Moscow and its environs (they were brought even at night to the BMW X5) and all sober installers. Critical equipment from the affected datahalls (affected by the lack of cold, not from a fire!) Was quickly transferred to other rooms, so that already at five in the morning they raised not only all their services, but also their customers.

We personally checked the room that “suffered” the most — except for the smell of burning and replaced raised floor tiles, it doesn’t remind of the fire at all (except for the equipment racks “temporarily” taken to another data hall, and the ones left there).

Status bar

Walking along the corridors, we see a touch screen with something very similar to the circuit of the data center halls. This is the status bar. They are stuck in the corridors in order, and they allow on-site engineers to quickly find, for example, the right air conditioning, well, they just report the current status.

Monitoring is built on Nagios (now there are several independent clusters) and covers all data centers (both OST and NORD). Additionally, there are applications for Android and iOS (for customers and employees), and a status page directly on the site.

On the display I notice an unusually high PUE - not everywhere in Russia it is considered to be considered, but still 1.90 is a bit much. The answer to this is purely economic - in 2012-2013, the racks in the data centers diverged like hot cakes, so there was no time for sentiment and energy efficiency. We put what is on the market, do it as quickly as possible and don’t care about these PUEs at all, because why? Even at the design stage, 80-90% of racks are bought, but the legislation in the Russian Federation about energy efficiency does not want to know anything. Although some commission in Amsterdam, Kondraty would have been enough from such numbers.

Nothing personal. Just a business.

Another feature is the labeling of data halls. On each site, data halls, in addition to the “official” alphanumeric coding, have “human” pseudonyms from the naval alphabet (Alpha Bravo-Charlie Delta ...). For some reason, all customers love Whiskey Hall.

There are no trifles in customer care

Such "fences" are set for some customers. We were not allowed to shoot in the mashrooms, so the DataLine © photo

DataLine customers are very large and respected companies, they really do not like it when photographing their equipment. Especially if the name (or logo) of the client is written directly on the counter in large letters. They do not like - and that’s it. Therefore, we shot the already famous walls with different inscriptions, and the engineer. The engineer can not be photographed either all or everywhere, but we tried. Since there are large and respected clients among the customers, they have specific requests - special racks (or even some kind of uber storage from several vendor racks), their fences, their ACS, their video surveillance and quick reaction group with assault rifles . In general, one of the unspoken (and vowels too) DataLine rules is to take care of the client and do well for him. Starting from vests and equipped workstations with Internet access and charging, and ending with various events for customers. The latter are also very useful on the other hand, that the same people, moving from company to company, bring them to DataLine. Just because everything is cozy, convenient and reliable. Business DataLine - is not only the delivery of rack spaces. It is also a cloud that is simultaneously certified for both 152- and PCI-DSS (we are going to make a podcast about it, follow the announcements), and providing connectivity (your fibers between the sites and from each of them to M9, your own optical ring for city, and so on until the last mile to the client’s office).

M - motivation

By the way, about inscriptions - ideas come not only from employees, but also from partners and customers, after which you can vote for the most successful on the official DataLine page in FB. Well, you can offer your idea there.

Meeting on the Elbe

Cross (photo provided by DataLine)

We stop at the Meet-Me-Room, where the operators are located. In MMR (there are three pairs of them in OST, according to the number of main optic inputs, for active and passive) connections from all data halls are reduced. In the cross, everything is boring and neat. But where the operator’s activity is, you recall the best years and shots from some BM18: wire noodles and miscellaneous equipment (DLink, Extreme, Juniper, Mikrotik ... and so on to ASR9k). There are more than fifty operators here, so everything is colorful. From MMR, optics and copper diverge along the halls. At least two tracks for each (options are possible), so you can not worry about the reliability of the connection. MMR is called "Elba", which is very subtle.

IBM Power n-nada?

Another find is the IBM P-series, which has never been turned on, and it has been standing in one of the corridors for several years. A car (along with a tape library and storage) was ordered by one of the customers, but "was not useful", so it was presented to a data center. Since then it’s worth it.

It’s good to be wealthy guys and give such presents, right? (Or maybe they didn’t give it, but forgot. Or they lost the cards. But it costs a long time)

Here is such a handsome man standing in NORD-4

We go to the energy center - three (or even four) DGUs for each data center (I remind you that there are four of them in OST). The diesel engine reaches full power in 30-40 seconds and produces 1900kVA. N + 1 redundancy, synchronization, etc. are included. The longest time spent on diesel engines in OST is 12 hours. in NORD - 2.5 days (after all, a storage of 85 tons is not for you khukh-mukhras).

Of course, there are contracts with different fuel companies for the supply of fuel. Switching to diesel occurs regularly, so that fuel is produced and contracts are checked. By the way, according to colleagues, at NORD when testing a diesel generator, which usually takes more than one hour, the grass manages to turn green even in winter at the place where the hot air from the diesel engines is emitted.

After a short bite to go to a NORD venue in a familiar coffee shop. This is also not one data center, but four independent buildings in one territory. Each has its own electricity inputs, its own diesel engines and its own cooling. NORD-1 and NORD-2 have been audited by Uptime Institute M&O. The freshest (and the best DataLine data center) is NORD-4, so we go there.

With a huge, it would seem, the number of data centers in Moscow, there is still a shortage. No, of course, one or two racks will be shipped to you without problems. Even twenty will be given. More is a question (and DataLine clients with such needs have not one or two). Specifically, there is a place (so far) in NORD-4, because one large wedge has moved to its own freshly built data center, but in other places it may no longer be so lucky. Nevertheless, now the NORD-5 design is in full swing, which they promise to launch in early 2020.

But back to our tour. Since we were traveling by car, we were launched into the gateway, in which machine gunners took us to the sight the number of our car was checked with the application, and all passengers were asked to go out to obtain passes (of course, according to the passport). For B is security.

NORD-4 is the pride of the company and the compilation of many years of experience in the construction and operation of data centers, it has Uptime Institute Tier III certificates for the design project and Facility (i.e., the data center built), as well as Operational Sustainability - the guys approached the matter as much as possible responsibly. Inside - four independent hermetic zones, two halls inside one building. Each has its own energy center (DGU, battery rooms) cooling circuits and fire extinguishing systems. General - optics inputs (there are two of them per building) and MMR.

Modding a healthy engineer

At the entrance to the data center, a coffee shop familiar to us is waiting for us (well, why not?). We rise to the monitoring center. Now there are two main ones - in OST and NORD-4 (in NORD-1,2,3 their duty shifts are also sitting). From each center you can watch both sites, so even here redundancy n + 1 is provided. On the video wall - we already know the dashboards in the light green range (of course, there are much more parameters on it than on the status display). We communicate a little bit for monitoring - no fashionable prometheuses and grafans, only Nagios, only hardcore and self-written visualization. For ten years (however, even more) of use, many custom checks and scripts have already been written, so moving out to something else looks like that adventure. About once a month, another company appears that offers to "quickly and painlessly" solve all problems, but so far it does not add up. By the way, the video wall also works on a self-assembled solution, which turned out to be several times cheaper than commercial analogues.

The local MCC

When you design, build and operate data centers for many years, you are not so much afraid of doing something yourself, rather than buying ready-made ones. DataLine, for example, does not hesitate to say that they, simply at the expense of scale, stepped on the maximum number of rakes, and each new data center is the son of difficult mistakes and experience that has been going on for more than ten years. Of course, each new data center has its own know-how, and, walking along NORD, you catch yourself thinking every now and then, people just took it and did it well. Not for the sake of, not for the sake of ratings and certificates - the usual engineering process, when at each iteration you grow above yourself and become better and better. An outside observer will say that it is boring and conservative, but the client, on the contrary, supports such stability with his own money. And in order to do well, it is not at all necessary to buy engineering systems or software for a lot, a lot of money (although without it, sometimes, in any way). More often decides experience, knowledge and engineering savvy.

For example, the NORD-4 building has 3.5 floors. The first three are full-fledged, with ceilings of 4.5 meters, where the halls are located. And the last one is technical, with a height of only 1.8 meters, and is given purely under the cooling system. Since the floor is technical, the requirements for it are much lower, which allowed us to simplify the project and save money.

View of the triangular data center

Max has a special kung fu not to fall for pictures

We finish the tour on the roof - it offers an excellent view of the nearby CHPP and the housing estate, as well as other NORD buildings. Two of them are triangular. On this occasion, colleagues even laughed that "DataLine is morons, for some reason they are building a triangular data center" To put there triangular racks and triangular servers. But nothing - they built and successfully operate, but at the same time they efficiently use the existing territory (which in Moscow, I must say, is not free). Another surprising (for me) fact is that most commercial clients want to be located in Moscow. A site for half a hundred kilometers along the Minsk highway (even if there are all the necessary capacities and communications) is already of little interest to anyone. So everything is nestled in non-rubber.

Cooling towers are no longer a data center, but a CHP

On this our next journey came to an end.

After there was still a long and productive conversation about joint plans and future podcasts (I hope you won’t have to wait long), so stay tuned for more interesting things soon.

Well, DataLine invites those who want to see everything with their own eyes on a tour. Completely free, but by appointment. Those who name the code word "linkmeup" are a plus in karma.

Stay tuned!