Clever Geek Handbook

Who are DevOps?

At the moment, this is almost the most expensive position in the market. The hustle and bustle around DevOps engineers exceeds all imaginable limits, much less with Senior DevOps engineers.

I work as the head of the integration and automation department, guess the English decryption - DevOps Manager. Whether the English decoding exactly reflects our daily activities is unlikely, but the Russian version in this case is more accurate. By the nature of my activity, it is natural that I need to interview future members of my team and, over the past year, about 50 people passed through me, and the same amount was cut off on the screen with my employees.







We are still looking for colleagues, because a very large layer of various kinds of engineers is hiding behind the DevOps label.







Everything written below is my personal opinion, you are not obliged to agree with it, but I admit that it will bring a touch to your attitude to the topic. Despite the risk of falling out of favor, I publish my opinion, because I believe that he has a place to be.







Companies differently understand who DevOps engineers are and for the sake of quickly hiring a resource they hang this label for everyone. The situation is rather strange, because companies are willing to pay unrealistic rewards to these people, receiving, in most cases, an admin tool for them.







So who are DevOps engineers?



Let's start with the story - Development Operations appeared as another step towards optimizing the interaction in small teams to increase the production speed of the product, as an expected consequence. The idea was to strengthen the development team with knowledge of procedures and approaches in managing the product environment. In other words, the developer must understand and know how his product works under certain conditions, it must understand how to deploy its product, which environmental characteristics to tighten to increase productivity. So, for some time, developers appeared with the DevOps approach. DevOps developers wrote build and packaging scripts to simplify their activities and the performance of a productive environment. However, the complexity of the decision architecture and the mutual influence of infrastructure components over time began to worsen the performance of environments, with each iteration, an ever deeper understanding of certain components was required, reducing the productivity of the developer himself due to the additional cost of understanding the components and tuning systems for a specific task . The developer’s own cost was growing, the cost of the product along with it, the requirements for new developers in the team jumped sharply, because they also had to cover the responsibilities of the “star” of development and, naturally, the “star” became less accessible. It is also worth noting that, in my experience, few developers are interested in the specifics of packet processing by the kernel of the operating system, packet routing rules, and host security aspects. The logical step was to attract an administrator who is familiar with this particular responsibility and assign the responsibility of a similar format to him, which, thanks to his experience, made it possible to achieve the same indicators at a lower cost in comparison with the cost of the development star. Such administrators were placed in a team and its main task was to manage test and productive environments, based on the rules of a particular team, with resources allocated to this particular team. So, in fact, DevOps appeared in the majority view.







Partially or completely, over time, these system administrators began to understand the needs of this particular team in the development field, how to simplify the life of developers and testers, how to roll out the update and not stay overnight at the office on Friday, fixing the deployment errors. Time passed, now system administrators became the “stars”, they understood what the developers wanted. In order to minimize impact, management utilities began to catch up, everyone remembered the old and reliable methods of isolating the OS level, which allowed minimizing security requirements, managing the network part, and the host configuration as a whole and, as a result, reduce the requirements for new "stars".







A “wonderful” thing has appeared - docker. Why wonderful? Yes, just because creating isolation in chroot or jail, as well as OpenVZ, required non-trivial knowledge of the OS, in a counter utility that allows you to simply create an isolated application environment on a certain host with everything you need inside and transfer the reins to the development again, and only manage the system administrator only one host, ensuring its security and high availability - a logical simplification. But progress does not stand still and systems are becoming more and more complicated, more and more components, one host no longer satisfies the needs of the system and it is necessary to build clusters, we again return to system administrators who are able to build these systems.







Cycle by cycle, various systems appear that simplify development and / or administration, orchestration systems appear that, exactly until you need to move away from the standard process, are easy to use. Microservice architecture also appeared to simplify everything described above - fewer relationships, easier to manage. In my experience, I did not find a fully microservice architecture, I would say 50 to 50 - 50 percent of microservices, black boxes, came in, processed, the other 50 - a torn monolith, services unable to work separately from other components. All this again imposed restrictions on the level of knowledge of both developers and administrators.







Similar "swings" of the level of expert knowledge of a particular resource continue to this day. But we were a little distracted, there are many points that are worth highlighting.







Build engineer / release engineer



Very highly specialized engineers who appeared as a means of standardizing software assembly processes and its releases. In the process of introducing the bulk Agile, it would seem that they ceased to be in demand, but this is far from the case. This specialization appeared as a means of standardization of the assembly and delivery of software on an industrial scale, i.e. using standard techniques for all products of the company. With the advent of DevOps, developers have partially lost their functions, since it was the developers who began to prepare the product for delivery, and given the changing infrastructure and the approach to the fastest delivery without regard to quality, over time they turned into a stopper of changes, since adherence to quality standards inevitably slows down deliveries. So, gradually, part of the Build / Release functionality of engineers migrated to the shoulders of system administrators.







Ops's so different



We move on and again the presence of a wide range of responsibilities and the lack of qualified personnel pushes us to tough specialization, like mushrooms after rain, various Operations appear:









DevOps - (in theory) a person who knows firsthand all the processes of the development cycle - development, testing, understanding the product architecture, is able to assess security risks, is familiar with automation approaches and tools, at least at a high level, and also understands pre and post product release support. A person is able to advocate both Operations and Development, which allows to build a favorable cooperation between the two pillars. Understanding team planning processes and managing customer expectations.







To perform this kind of work and responsibilities, this person must have the means to control not only the development, testing, but also the management of the product infrastructure, as well as resource planning. DevOps in this sense cannot be located either in IT, or in R&D, or even in PMO, it must have influence in all these areas - the technical director of the company, Chief Technical Officier.







Is this so in your company? - I doubt it. In most cases, it is either IT or R&D.







The lack of funds and the possibility of influencing at least one of these three lines of business will shift the weight of the problems to the side where these changes are easier to apply, such as applying technical restrictions on releases due to dirty code according to the data of the static analyzer systems. That is, when the PMO sets a tight deadline for the release of functionality, R&D cannot give a high-quality result within these deadlines and gives it as it can, leaving refactoring for later, DevOps related to IT, by technical means blocks the release. The lack of authority to change the situation, in the case of responsible employees, leads to the manifestation of hyperresponsibility for what they cannot influence, moreover, if these employees understand and see mistakes, and how to fix them is “Happiness in ignorance”, and as a result burnout and loss of these employees.







Market DevOps Resources



Let's look at a few DevOps job openings from different companies.







We are ready to meet with you if you:

  1. Own Zabbix and know what Prometheus is;
  2. Iptables;
  3. Graduate Student BASH;
  4. Professor Ansible;
  5. Linux guru
  6. Know how to use debug and find application problems together with developers (php / java / python);
  7. Routing does not cause you tantrums;
  8. Pay significant attention to system security;
  9. Backup “everything and everything”, and also successfully restore this “everything and everything“;
  10. You know how to configure the system so as to squeeze out of the minimum - the maximum;
  11. Configure bedtime replication on Postgres and MySQL;
  12. Setting up and adjusting CI / CD for you is a necessity like breakfast / lunch / dinner.
  13. Have experience with AWS;
  14. Ready to develop with the company;


So:









Summarizing this vacancy, we can say that the Middle / Senior System Administrator is enough for the guys.







By the way, do not strongly separate the admins on Linux / Windows. Of course, I understand that the services and systems of these two worlds are different, but everyone has one basis and any admin who respects himself is familiar with one or the other, and even if he is not familiar, it will not be difficult for a competent admin to get acquainted with this.







Consider another vacancy:







  1. Experience in building highly loaded systems;
  2. Excellent knowledge of the Linux OS, system-wide software and the web stack (Nginx, PHP / Python, HAProxy, MySQL / PostgreSQL, Memcached, Redis, RabbitMQ, ELK);
  3. Experience with virtualization systems (KVM, VMWare, LXC / Docker);
  4. Knowledge of scripting languages;
  5. Understanding the principles of operation of networks of network protocols;
  6. Understanding the principles of building fault-tolerant systems;
  7. Independence and initiative;


Parse:









Summarizing - Middle / Senior System Administrator







One more:







  1. Experience devops;
  2. Experience in using one or more products to form CI / CD processes. Gitlab CI will be an advantage;
  3. Work with containers and virtualization; If you used docker - fine, but if k8s - great!
  4. Experience in an agile team;
  5. Knowledge of any programming language;


We'll see:









I would also like to leave a remark regarding 3 points in order to strengthen the understanding of why this point is covered by the system administrator. Kubernetes is just an orchestration, a tool that wraps direct commands to network drivers and virtualization / isolation hosts in a couple of commands and allows you to make communication with them abstract, that's all. For example, take the 'build framework' Make, which, by the way, I do not consider as a framework. Yes, I know about the fashion of shoving Make anywhere, where it is necessary and not necessary - to wrap Maven in Make, for example, seriously?

In essence, Make is just a wrapper over the shell, which simplifies the compilation, linking, compilation environment, as well as k8s.







Once, I interviewed a guy who used k8s in his work on top of OpenStack, and he talked about how to deploy services on it, however, when I asked about OpenStack, it turned out that it is being administered, as well as being raised by system administrators. Do you really think that the person who raised OpenStack, regardless of which platform he uses behind him, is not able to use k8s? =)

This applicant is actually not DevOps, but the same System Administrator and, to be more precise, Kubernetes Administrator.







We summarize again - Middle / Senior System Administrator will be enough for them.







How much to hang in grams



The spread of the offered salaries for the indicated vacancies is 90k-200k

Now I would like to draw a parallel between the monetary rewards of System Administrators and DevOps Engineers.







In principle, to simplify, you can scatter grades of experience, although this will not be accurate, for the purposes of the article is enough.







Experience:







  1. under 3 years old - Junior
  2. under 6 years old - Middle
  3. more than 6 - Senior


The employee search site offers:

System Adminsitrators:







  1. Junior - 2 years - 50k rub.
  2. Middle - 5 years - 70k rubles.
  3. Senior - 11 years old - 100k rubles.


DevOps Engineers:







  1. Junior - 2 years - 100k rub.
  2. Middle - 3 years - 160k rub.
  3. Senior - 6 years old - 220k rub.


According to the experience of “DevOps”, we used the experience, at least somehow affecting the SDLC.







From the above it follows that in fact, companies do not need DevOps, and also that they could save at least 50 percent of the originally planned costs by hiring an Administrator, moreover, they could more clearly determine the responsibilities of the person being sought and quickly close the need. Do not forget that a clear division of responsibilities reduces the requirements for personnel, as well as create a more favorable atmosphere in the team, due to the absence of intersections. The vast majority of vacancies are full of utilities and DevOps labels, however, they do not really have the requirements for DevOps Engineer, they are just requests for the admin administrator.







The process of training DevOps engineers is also limited only by a set of specific works, utilities, does not give a general understanding of processes and their dependencies. It’s certainly good when a person can use Terraform to deploy AWS EKS, in conjunction with the Fluentd side-car in this cluster and the AWS ELK stack for the logging system in 10 minutes, using only one command in the console, but if he does not understand the principle of processing logs and why they are needed, not to know how to collect metrics on them and track the degradation of the service, then it will be the same enikey, able to use some utilities.







Demand, however, creates supply, and we see an extremely overheated market for DevOps positions, where the requirements do not correspond to the real role, but only allow system administrators to earn more.







So who are they? DevOps or greedy system administrators? =)







How to live further?



For employers - it is more accurate to formulate requirements and look for exactly those who are needed, and not scatter labels. You do not know what DevOps do - you do not need them in this case.







Workers - Learn. Constantly improve your knowledge, look at the overall picture of the processes and track the path to the goal. You can become whatever you want, you just have to try.







More articles:


All Articles