How we transplanted the whole team into another language in one day (actually not)

Beginning of a template for quick "grounding" of PHP developers in Go



For 15 years we have been doing a backend in PHP. And then one day a strategic decision was made: first, rewrite the most heavily loaded places on Go, and then develop new services on it.



Imagine: you want to talk about a new language to a team of 40 developers who prepare PHP so well that they put together a multi-threaded real-time and high-availability system. In the worst case, they will burn you, at best they will listen, but they will continue to do as before. This is if you enter the language by force.



Therefore, we decided to first arrange a hackathon, and then invite everyone to the dark side. If at least ten people had recruited them, then over time they would have shown everyone else a personal example of how cool this is.



I’ll tell you in steps how to repeat such an experience with you. And about the results of our implementation. And about what happens in the head of a PHP developer when he sees a new language.



We started with the preparation of the template and CI / CD, which allows you to shut down in 15 seconds. To write it yourself, you need about two weeks. We made it in advance.

Step 1. Tell about Golang

We often have a lot of internal seminars. Someone wants to tell something, writes an ad, gathers people and tells. It happens that it’s especially the regulation of air traffic in Russia, it happens that it is a story about travel, but much more often we are talking about some kind of technology or feature that was implemented. And it may be useful to someone else. So, we put together an internal seminar on Go, where we talked about the advantages of the language. Fast (compiled without a tambourine: PHP can also be compiled, but more difficult), multi-threaded out of the box, quite simply supported, you can write without a framework directly in a notebook, that is, there is no hemorrhoids with third-level dependencies that the developed framework drags along with it. Deploy in 15 seconds. And it seems that there are not even any obvious flaws in our situation. We suggest to try. It was like a story about another world: here is the language, look how it differs, next weekend there will be a hackathon for those who are interested.

Step 2. Documentation

Three days before the hackathon, we send a link to a half-hour introductory course on syntax, basic data types, and functionality. And basic documentation. Here's what the developers got:



golang.org/doc

tour.golang.org



Plus Go-senior contacts inside.

Step 3. Groups

Further, we proposed to break into groups and bring tasks that we would like to implement here and now in a new language. They immediately warned that the hackathon would be in the office after hours (on Saturday), the matter was voluntary, but it would be possible to solve the combat task and learn in practice in almost pair mode with a person who knows the nuances of the language. This interested 40 people, and they began to fight in groups and recruit tasks. Not everyone succeeded with the groups: many came with the task of only one, the groups were two, less often - three people. The tasks themselves were on top of backlogs or related to duplicating some kind of braking space.



The ultimate goal was to make a task at the hackathon, which then can be rolled up a little and “sold” to the product owner (direction manager). In the spirit: "Look, it worked before 40 seconds, and now it does everything in two." This part is the most important for the strategic success of introducing the language in teams: you need a small victory at the very beginning.

Step 4. Invisible work

We prepared a template, about which I spoke above, for our infrastructure, which would allow you to run immediately without dancing to build and connect everything you need. This template replaces the framework. Spread it does not make much sense: I repeat, it is very imprisoned specifically for us.



I will list only the functionality that was laid down in the template:

• Integration with k8s.
• Export metrics to prometheus.
• Configuration through environment variables.
• Http server.
• Opentracing.
• Logger

We have been working on the study of Kubernetes for a long time and as a solution we chose OpenShift, which offered great opportunities for customizing and configuring CI / CD. So we sharpened the template immediately under k8s. The created application in OpenShift creates the repository, collects the application skeleton, the service in k8s, applies security policies, collects the binary, deploys in k8s and in less than a minute the service is available on the network. So the hackathon also became an event where developers were introduced to our future PaaS.



We also prepared a dashboard in Grafana with basic metrics, set up export logs for Kibana and agents for Jaeger. For each service, links to services are generated, on which you can immediately get all the necessary information.



Next, it was necessary to agree on access to the databases that are needed for solving problems by hackathon participants. This is wrong, but we connected to the military bases in read-only for the duration of the work. This is another reason why Saturday is the smallest load on this day.



They warned the kitchen (I remind you that we have it free for snacks - cottage cheese-fruits, but not for hot food) about the event. Food was brought to the office so that we would not be distracted.

Step 5. Hackathon itself

For obvious reasons (family, relaxation), some failed on Saturday, so we divided 15 people into calls.



They talked about our PaaS platform, how to create an application from a template, how to use a template. This framework, in which there are wrappers for collecting metrics, health checks and metrics, connecting to a database and so on, showed how to connect all kinds of different things. It was possible to remove everything in place or in advance (the template was also handed out to the hackathon). This is a ready-made application that needs to be expanded.



They talked about deployment and development features for Kubernetes (configuration, logging, metrics, working with ready-samples, resource allocation for the processor and memory, etc.). They tried to deploy "Hello, world!".







The guys themselves created the services. I was pleasantly surprised by the simplicity and usability of the template. Someone already had time to practice on Golang, someone just leafed through the documentation.







We did not have a goal to show how to write well, we had to get a rough functional code and show how everything can quickly go to the preprod, how everything works quickly and how easy it is to make edits.



The plan was this: the teams write, if you have questions, we answer. Then a review session and help on how to do something more correct in architecture on Golang.



Of the cognitive distortions of PHP middle, it is worth noting the “old” way of thinking in one thread. Go has a good built-in competitive model and multithreading out of the box, which was not in PHP. The code turns out to be very readable, which also few expected.



In practice, we do not always use multithreading. Depends on the task. But under the hood, it is also there, and, where necessary, the speed increases due to it. For example, a separate goroutine is created for any incoming REST request. Golang can process several requests at the same time, when most other languages ​​(including PHP), by default, queue them and process them in one thread for each worker.



The second feature is a big pain with the return of errors. If multithreading is just another architecture and it is easy to get used to it in an hour or two, then in the case of errors when switching from PHP, this is a change in the whole model of thinking. Returning errors from functions is the main paradigm in Go. In PHP, try-catch usually wraps things around. And here it is required to process explicitly in the function body. Some people were brutally bombed by this “slowdown”, they actively resisted. That's because the approach is pretty close to TDD, and TDD requires patience while writing code, but it’s easy to debug. This increases the code line by line "if - an error", but once you write and do not return. This is very important in the architecture of microservices, because microservices grow like mushrooms and one developer can write dozens of microservices. Constantly switching contexts and debugging errors is very expensive and inefficient. If everything is done correctly, then the application will work for years, as effectively as during development. And in PHP in the same situation, it is likely that if the code is written incorrectly, new errors will constantly pop up, after which it will be necessary to debug everything again.



It turned out footcloths in main. When the functional part was launched, they already figured out how best to break it down into packages. Then there was a review. When Go-senior helps you, it is very useful right away. Someone was shown how to replace constant visits to the base with a cache in RAM. Someone was helped to rewrite the code in parallel execution. Somewhere there were small plugs, like the fact that the HTTP server in PHP traditionally rises and dies, but here it remains to hang.

Step 6. Repository

At the end of the hackathon, the code was folded into the repository, where everyone could see the code of the neighboring teams and send a pull request. This is also part of the training, and during the week these requests went quite actively.

Summary

Eight hours of hackathon in two streams of plus or minus 15 people (I remind you, out of 70 developers, 40 wanted to try Go and received documentation and a template, but not everyone could participate in one day or wanted to participate physically).



A quarter of the services gave the expected result immediately. Another part either required long modifications on the part of the PHP monolith, or simply was more than eight hours wide. About half of the services were completed before the sale later.



As a result, everyone got acquainted with the language, the OpenShift platform, how to marry them together, and so on. We saw the application template and the new CI / CD, got the whole set of tools to further solve our problems on Go.



Then the features were “sold” to product owners. We answered questions on architecture from team leaders.



Over the next month, almost all together added a template (which is a mini-framework) to the needs of departments. As a result, he became bigger. Yes, another feature of the template: the whole configuration - through environment variables, that is, with local development and when switching to preprod-prod, everything works perfectly equally. Docker is good.



Not only Go went from the hackathon to the masses. Templates appeared in the PaaS platform for PHP and Node.js, but all the same, all other things being equal, developers more often choose Go as the language for microservice. As a result, in practice, all new selling microservices written after the hackathon are written specifically on Go. That is, we cut a monolith and immediately write everything new under Kubernetes. By the end of the year we will move there almost completely.



Those who did not come asked their colleagues what it was. Then they approached and asked for the language, did some tasks in the background (to the main work), then put the code in the repository and received pull requests from other just trained developers and from Go-specialists. Interested in books. That is, we were still a few weeks of technical support of the second line.



For those who wanted to continue their studies, they created a group in Telegram with books and more. We also organized a guild of Golang developers, where we weekly discuss emerging issues in the work, develop common components, share good practices. Any developer can enter.



The biggest consequence is that after a month we rewrote one of the busiest sections, and it began to work about six thousand times faster in production. But I’ll better talk about this separately: there are not only features of the language, but also the implementation of multithreading, a couple of architectural improvements and a bit of shamanism. The most important thing is our experience of learning the language, and it is generally easy to repeat it if you have at least one Go-specialist who will spend a couple of weeks learning.



Other posts about our development: the history of the 15-year-old monolith , a grocery approach to the kitchen , what the developer should know about the business , and especially the return of airline tickets .