How to migrate to the cloud in two hours thanks to Kubernetes and automation

URUS tried Kubernetes in different forms: a standalone bare metal deployment on Google Cloud, and then moved its platform to Mail.ru Cloud Solutions (MCS). How to choose a new cloud provider and how to manage to migrate to it in a record two hours is told by Igor Shishkin ( t3ran ), senior system administrator of URUS.

What does Urus do

There are many ways to improve the quality of the urban environment, and one of them is to make it environmentally friendly. This is exactly what the URUS - Smart Digital Services company is working on. It implements solutions that help enterprises control important environmental indicators and reduce their negative impact on the environment. Sensors collect data on air composition, noise level and other parameters, and then send them to a single platform “Urus-Ekomon” for analysis and preparation of recommendations.

How is the work of "Urus" inside

A typical client of Urus is a company located in or near a residential area. It can be a factory, port, railway depot or any other facility. If our client has already received a warning, was fined for environmental pollution, or wants to make less noise, reduce the amount of harmful emissions, he comes to us, and we already offer him a turnkey solution for environmental monitoring.









A graph of H ₂ S concentration monitoring shows regular nightly emissions of a nearby facility



The devices that we use in URUS contain several sensors that collect information about the content of certain gases, noise levels and other data for assessing the environmental situation. The exact number of sensors is always determined by a specific task.









Depending on the specific measurement, devices with sensors can be located on the walls of buildings, poles and other arbitrary places. Each such device collects information, aggregates it and sends it to the data receiving gateway. There we save data for long-term storage and pre-process for subsequent analysis. The simplest example of what we get after analysis is an air quality index, aka AQI.



At the same time, many other services work on our platform, but basically they are of a service nature. For example, the notification service sends notifications to customers if any of the monitored parameters (for example, the content of CO ₂ ) has exceeded the permissible value.

How do we store data. The story of Kubernetes on bare metal

The URUS eco-monitoring project has several data warehouses. In one we hold the "raw" data - what we received directly from the devices themselves. This storage is a “magnetic” tape, as on old cassettes, with a history of all indicators. The second type of storage is used for pre-processed data - data from devices enriched with metadata about sensor connections and device readings, affiliation with organizations, locations, etc. This information allows you to dynamically evaluate how a particular indicator has changed over a certain period of time . We use the raw data storage, including as a backup, and to restore pre-processed data, if such a need arises.



When we looked for ways to solve the storage problem several years ago, we had two options for choosing a platform: Kubernetes and OpenStack. But since the latter looks pretty monstrous (just look at its architecture to see this), then we settled on Kubernetes. Another argument in his favor was a relatively simple program control, the ability to more flexibly cut even iron nodes into resources.



In parallel with the development of Kubernetes itself, we also studied ways of storing data, while we kept all our storages in Kubernetes on our hardware, we received excellent expertise. Everything that we had then lived on Kubernetes: statefull storage, monitoring system, CI / CD. Kubernetes has become our all-in-one platform.



But we wanted to work with Kubernetes as a service, and not engage in its support and development. Plus, we didn’t like how much its content on bare metal cost us, and development was always required for us! For example, one of the first tasks was to integrate Kubernetes Ingress controllers into the network infrastructure of our organization. This is a cumbersome task, especially if you imagine that at that time nothing was ready for programmatic resource management like DNS records or IP allocation. Later we started experimenting with an external data warehouse. We didn’t get to the implementation of the PVC controller, but even then it became clear that this was a big field of work, for which it was necessary to single out individual specialists.

Migrating to the Google Cloud Platform Temporary Solution

We realized that this could not continue further, and transferred our data from bare metal to the Google Cloud Platform. In fact, then for the Russian company there were not many interesting options: in addition to the Google Cloud Platform, only Amazon offered a similar service, but we still settled on a solution from Google. Then it seemed to us economically more profitable, closer to Upstream, not to mention the fact that Google itself is a kind of PoC Kubernetes in Production.



The first serious problem appeared on the horizon in parallel with how our customer base grew. When we had a need to store personal data, we were faced with a choice: either we work with Google and violate Russian laws, or we are looking for an alternative in the Russian Federation. The choice, in general, was predictable. :)

How we saw the perfect cloud service

By the beginning of the search, we already knew what we wanted to get from the future cloud provider. What service we were looking for:

• Fast and flexible . So that we can quickly add a new node or deploy something at any time.
  
  
• Inexpensive . We were very worried about the financial issue, since we were limited in resources. We already knew that we wanted to work with Kubernetes, and now the task was to minimize its cost in order to increase or at least maintain the effectiveness of using this solution.
  
  
• Automated . We planned to work with the service through the API, without managers and phone calls or situations where you need to manually raise several tens of nodes in emergency mode. Since most of our processes are automated, we expected the same from a cloud service.
  
  
• With servers in the Russian Federation . Of course, we planned to comply with Russian law and the very 152-FZ.
  
  

At that time, Kubernetes aaS providers were few in Russia, while choosing a provider, it was important for us not to compromise our priorities. The Mail.ru Cloud Solutions team, with which we have begun to work and are still working, has provided us with a fully automated service with API support and a convenient control panel in which there is Horizon - with it we could quickly raise an arbitrary number of nodes.

How we managed to migrate to MCS in two hours

In such relocations, many companies are faced with difficulties and failures, but in our case they were not. We were lucky: since we were already working on Kubernetes before the start of the migration, we just fixed three files and launched our services on a new cloud platform, in MCS. Let me remind you that by that time we had finally left bare metal and lived on the Google Cloud Platform. Because the move itself took no more than two hours, plus a little more time (about an hour) it took to copy data from our devices. Then we already used Spinnaker (a multi-cloud CD service to provide Continous Delivery). We also quickly added it to the new cluster and continued to work as usual.



Thanks to the automation of development processes and CI / CD, Kubernetes in URUS is occupied by one specialist (and this is me). At some stage, another system administrator worked with me, but then it turned out that we had already automated the entire main routine, and from the side of our main product there were more and more tasks and it made sense to direct resources to this.



We received from the cloud provider what we expected, since we started cooperation without illusions. If there were any incidents, then mostly technical ones and those that are easily explained by the relative freshness of the service. The main thing is that the MCS team quickly eliminates flaws and quickly responds to questions in instant messengers.



If we compare the experience with the Google Cloud Platform, then in their case I did not even know where the feedback button is, since it simply was not necessary. And if any problems happened, Google itself sent out notifications unilaterally. But in the case of MCS, a big plus, I believe that they are as close to Russian customers as possible - both geographically and mentally.

As we see working with clouds in the future

Now our work is closely tied to Kubernetes, and it completely suits us in terms of infrastructure tasks. Therefore, we do not plan to migrate from it somewhere, although we are constantly introducing new practices and services to simplify routine tasks and automate new ones, increase the stability and reliability of services ... Now we are launching the Chaos Monkey service (specifically, we are using chaoskube, but this does not change the concept: ), which was originally created in Netflix. Chaos Monkey does one simple thing: at arbitrary time it deletes an arbitrary sub in Kubernetes. This is necessary for our service to live normally with the number of instances n – 1, so we accustom ourselves to be prepared for any malfunctions.



Now I see the use of third-party solutions - the same cloud platforms - as the only right one for young companies. Usually at the beginning of the journey they are limited in resources, both human and financial, and building and maintaining your own cloud or data center is too expensive and time-consuming. Cloud providers can minimize these costs, they can quickly get the resources necessary for the services to work here and now, and pay for these resources in fact. As for the Urus company, for now we will remain faithful to Kubernetes in the cloud. But who knows, maybe we will have to expand geographically, or implement solutions based on some specific equipment. Or maybe the amount of resources consumed will justify its own Kubernetes on bare metal, as in the good old days. :)

What we learned from our experience with cloud services

We started using Kubernetes on bare metal, and even there it was good in its own way. But its strengths were revealed precisely as an aaS-component in the cloud. If you set a goal and automate everything as much as possible, you will be able to avoid vendor lock-in and moving between cloud providers will take a couple of hours, and the nerve cells will remain with us. We can advise other companies: if you want to start your (cloud) service, having limited resources and maximum speed for development, start right now by renting cloud resources, and build your data center after Forbes writes about you.