TeamCity Recipes. Report Yandex.Taxi
My name is Eduard Matsukov, I'm making a Taximeter - an application for Yandex.Taxi drivers. I am engaged in infrastructure and everything connected with it. Some time ago I made a report - I talked about the experience of TeamCity friendship with our project and with developers in general. A separate part of the report is devoted to what Kotlin has to do with it.
- Almost every day they come to me personally and to our developers with questions. And where to get the assembly? And where to get such a branch? Why did something fall? Where is the problem in my code? Why is something not working correctly? To do this, we have a lot of self-written infrastructure in the project, plugins, various hacks and tricks that we use. On the one hand, to make life easier for the developer, on the other, to implement specific business tasks.
And at some point, of course, we use CI and TeamCity as well. We got confused - we taught TeamCity to be friends with Kotlin and brought, we can say, the entire CI and the entire assembly to a whole new level.
But first, a little history - to understand how we came to this and why this level I call a separate canon. TeamCity has existed in Yandex for many years. We had to live in this shared server, where the entire backend, the whole frontend, and, more recently, all mobile applications are hosted. About two years ago we all came together. And each developer sets up each project not even as he wants, but as he can or as far as he understands how much he wants to understand the system. And there is no one person who knows everything and knows how. Few people want to bother, study separately templates, TeamCity wilds. Therefore, everyone is sawing, who is what much.
We lived in this single server, and last year we had an accident at TeamCity. About a week was complete downtime. Assemblies were not collected, testing was constantly complaining. Someone contrived, collected locally.
This is due to the fact that our TeamCity server was, roughly speaking, a knee-high solution that suddenly grew into a great service. It is used by thousands of developers on Yandex. Of course, there was some kind of fault tolerance, but it also refused. The next time TeamCity was updated after the restart, it turned out that several hard drives had simply collapsed, and we could not rise again. I had to get out.
We need to draw conclusions from everything that happened. And we, of course, made these conclusions: we analyzed why this happened and how to make sure that this did not happen again.
First of all, it’s important that we went up for a very long time and restored our service. By service, I mean both a technical process and partly a business process for the banal delivery of releases, for the assembly of pool requests. We lost a lot of artifacts, including release builds, we lost a lot of time on pool requests, on the fact that testing was not able to do its job properly. And of course, we spent quite a bit of time restoring the project from scratch, reconfiguring the entire structure, the entire build system. And then we realized that it was time to change something and set up our own server.
We went to this for a long time. Not to say that only one accident led to this conclusion. In general, we decided that it was time to go to the mountain, to do this whole thing ourselves. We started a service rollout. This is done extremely quickly: a couple of days and you're done. When you unfold all this yourself and you can dig into the inside, administer a little, then interesting features are striking. One of them - the new TeamCity allows you to configure versioning.
Versioning is very primitive, but at the same time very reliable, beautiful and cool. Everything that is stored in TeamCity regarding your or any other project can be safely uploaded to Git, and you can live on it happily. But there are a couple of problems.
The first problem is that all people are used to working with TeamCity exclusively through the interface, and this habit is hard to eradicate. There is a small life hack here: you can simply prohibit any changes from the interface and make all people relearn. Our team has 2,000 developers. Not a good way, right?
In fact, the cons end there. The most important minus is that people have to relearn for something new. So, they need to be given ground in order to make a personal conclusion about why this is necessary at all. And this is necessary then, because TeamCity, thanks to versioning, does not allow you to apply changes that somehow break the system. TeamCity itself falls back on the latest stable revision.
In TeamCity, you can start each project for this versioning and configure it quite flexibly.
A little bit of educational program. All projects in TeamCity are arranged in a tree. There is some kind of common root, and further from it comes such a simple structure. Each project is the top of this graph. It can act as a certain set of configurations that build something, as well as a parent for other projects.
In Git, you can ship either all at once, or a specific piece. For example, if colleagues from the backend with the frontend do not want to use versioning, please, you can not count on them, and simply secure your personal project.
You can set up a rather complex hierarchical system, which our team eventually came to. We have one common big root and some small roots. Backend, mobile development, frontend, Yandex.Food - they all live each in their own separate repository. At the same time, information about all these projects is stored in a large common repository - in the root.
After you finally connect this versioning, install it with all your colleagues, where who will live and how, who will be engaged in support - after all this, you have to make a difficult choice.
TeamCity supports only two formats of configs. With XML, I suspect no one will want to work, so we chose the second format. It allows you to make these configs on a Kotlin script.
eamCity creates a maven-project, some semblance of any ordinary project. You can do one of two things with it: either upload to your project - Android, backend, it doesn’t matter - or leave it as a standalone project. Then you will have an independent repository with an independent project.
What is the plus of this approach? Personally, I and those guys who deal with our infrastructure on the backend and frontend, immediately bribed something. And even those who are not familiar with Kotlin, who heard about it for the first time, went and began to teach him.
These two lines create the whole project. This is the dialect of the TeamCity API. The API changes every major version. There are 2018-2, 2018-1, 2017, etc. Soon, hopefully, the 2019th will be released.
The second line simply declares the project.
Here is the project itself. This is absolutely real code. That's what our root repository looks like now. Nothing extra, nothing complicated. The only manual work that is required here is to manually create the UUID yourself. TeamCity requires that each object, each project has its own unique identifier. You can write anything there. I just use the standard nickname uuidgen team.
Here begins the adventure in the Kotlin DSL. I think this is a completely uncomplicated language for mastering. By uploading it to IDEA, Eclipse or any other IDE, you can get all the documentation, highlighting, autocomplete, tips. In fact, many of them are missing in the interface. Therefore, my personal experience says that working with the code is much more convenient, simpler and more intuitive. We are still developers.
Something like this looks like a real config now working at the same time, supporting TeamCity configs themselves. That is, TeamCity builds its own configs in its own environment. If all is well and everything has gone wrong, he calmly ships it in memory and replicates the changes to PostgreSQL. The base is already connected to the service itself. And here it will be a sin not to use all the features of Kotlin.
In this case, unlike XML, these configs can be described using polymorphism, inheritance - any features of the Kotlin language are allowed. The only important point is that all this can eventually turn into chaos that existed with us before we introduced versioning of the configs on the Kotlin script.
But, oddly enough, this chaos has become much less. Because before it was not quite obvious how to do what I want, how to achieve this or that feature? From the code, in my practice, it’s much easier to understand how to implement any feature.
The most interesting adventures begin here: how do we implement some things and how, in principle, make the interaction of the project with TeamCity easier?
Everyone here present in one form or another is preparing a release, participating in its assembly, in the publication. We publish our releases on various channels on Google Play.
We have beta, there are experiments, there is stable. We use a special plugin with a robot that posts comments with a report on the release build in the release ticket. And all this is set up with such a beautiful window. It appears as soon as you try to collect the release. These questions cannot be avoided.
From the TeamCity interface, it looks something like this. To immediately understand what, where, where and how, you need to read through each parameter, you need to experiment. From the documentation, in addition to what is visible on the screen, nothing else can be gleaned.
In code, it looks like this. At least so far, for half a year, no one has yet come and asked - how can I make some feature? Most often from the code it is all intuitively clear.
At the same time, some things are done quite simply, but hidden behind several layers of the interface. I have to walk, go back and forth.
Here is an example of how security is implemented in TeamCity. In my practice, for most people, TeamCity seems to be a fairly simple cold system that out of the box does not support integration, for example, with security services. Therefore, all tokens, all keys, all credentials with us most often stuck out in the open. Why not?
In fact, TeamCity is safe. He can create his own special file on his server, which is called - credential json, as shown. And he creates such a key for each token, for each credential, which we specially generate through the interface. You can already put it into the code and be sure that this credential will never come up in TeamCity logs or in the TeamCity interface. The system can cut these keys literally from anywhere. The whole interface is, roughly speaking, a kind of decoration.
Okay, we’ve set up some of our parameters, made a forward of the required parameters, for example, to build the release. But what if we want to go further? And we wanted to go further. During assembly, a lot of different steps are launched. We run several nested libraries that build from completely different repositories. And we just wanted to pull up fresh changes. All that is now. Do not bother - for example, do not collect an auxiliary library for pool requests, do not upload it to the maven repository, do not add additional gestures to the pool request.
We just set up the chain assembly. I will show to the end how unobvious and inconvenient it is to do this from the interface, in my personal opinion. And there already judge for yourself.
The chain assembly in the interface looks something like this.
It looks something like this in code. We simply indicate which particular configuration is dependent and what to do if any of the configurations did not work or was canceled by the user from the outside. In this case, I do not want the assembly to start at all. Because what is the point of it if we have not collected all the dependent libraries?
Around the same spirit, all other things are being done. And the entire project in TeamCity takes literally 500 lines of code.
It turns out that you can forward some interesting parameter through all the dependencies. I showed chaining for a reason. Chains is convenient, but hard to prepare in the interface. And TeamCity does not document such an important feature as forwarding through parameters. What is it for? Suppose, in our assembly in Gradle or somewhere else, we want to get tied to some specific field, forward the same address to the release ticket. And we want to do this once, and not for each nested assembly.
TeamCity has a not-so-obvious and completely non-documented parameter - reverse.dep (reverse dependency). It throws all of the parameters that come after the asterisk into all nested builds.
At the output, we get about such a simple structure. You can complicate it and make nesting as deep as your imagination or needs. And to be sure that in all these dependencies, in all these configurations, all of our parameters that we expect at each step of the assembly will be forwarded. Ready to answer your questions. Thanks to all!
- Almost every day they come to me personally and to our developers with questions. And where to get the assembly? And where to get such a branch? Why did something fall? Where is the problem in my code? Why is something not working correctly? To do this, we have a lot of self-written infrastructure in the project, plugins, various hacks and tricks that we use. On the one hand, to make life easier for the developer, on the other, to implement specific business tasks.
And at some point, of course, we use CI and TeamCity as well. We got confused - we taught TeamCity to be friends with Kotlin and brought, we can say, the entire CI and the entire assembly to a whole new level.
But first, a little history - to understand how we came to this and why this level I call a separate canon. TeamCity has existed in Yandex for many years. We had to live in this shared server, where the entire backend, the whole frontend, and, more recently, all mobile applications are hosted. About two years ago we all came together. And each developer sets up each project not even as he wants, but as he can or as far as he understands how much he wants to understand the system. And there is no one person who knows everything and knows how. Few people want to bother, study separately templates, TeamCity wilds. Therefore, everyone is sawing, who is what much.
We lived in this single server, and last year we had an accident at TeamCity. About a week was complete downtime. Assemblies were not collected, testing was constantly complaining. Someone contrived, collected locally.
This is due to the fact that our TeamCity server was, roughly speaking, a knee-high solution that suddenly grew into a great service. It is used by thousands of developers on Yandex. Of course, there was some kind of fault tolerance, but it also refused. The next time TeamCity was updated after the restart, it turned out that several hard drives had simply collapsed, and we could not rise again. I had to get out.
We need to draw conclusions from everything that happened. And we, of course, made these conclusions: we analyzed why this happened and how to make sure that this did not happen again.
First of all, it’s important that we went up for a very long time and restored our service. By service, I mean both a technical process and partly a business process for the banal delivery of releases, for the assembly of pool requests. We lost a lot of artifacts, including release builds, we lost a lot of time on pool requests, on the fact that testing was not able to do its job properly. And of course, we spent quite a bit of time restoring the project from scratch, reconfiguring the entire structure, the entire build system. And then we realized that it was time to change something and set up our own server.
We went to this for a long time. Not to say that only one accident led to this conclusion. In general, we decided that it was time to go to the mountain, to do this whole thing ourselves. We started a service rollout. This is done extremely quickly: a couple of days and you're done. When you unfold all this yourself and you can dig into the inside, administer a little, then interesting features are striking. One of them - the new TeamCity allows you to configure versioning.
Versioning is very primitive, but at the same time very reliable, beautiful and cool. Everything that is stored in TeamCity regarding your or any other project can be safely uploaded to Git, and you can live on it happily. But there are a couple of problems.
The first problem is that all people are used to working with TeamCity exclusively through the interface, and this habit is hard to eradicate. There is a small life hack here: you can simply prohibit any changes from the interface and make all people relearn. Our team has 2,000 developers. Not a good way, right?
In fact, the cons end there. The most important minus is that people have to relearn for something new. So, they need to be given ground in order to make a personal conclusion about why this is necessary at all. And this is necessary then, because TeamCity, thanks to versioning, does not allow you to apply changes that somehow break the system. TeamCity itself falls back on the latest stable revision.
In TeamCity, you can start each project for this versioning and configure it quite flexibly.
A little bit of educational program. All projects in TeamCity are arranged in a tree. There is some kind of common root, and further from it comes such a simple structure. Each project is the top of this graph. It can act as a certain set of configurations that build something, as well as a parent for other projects.
In Git, you can ship either all at once, or a specific piece. For example, if colleagues from the backend with the frontend do not want to use versioning, please, you can not count on them, and simply secure your personal project.
You can set up a rather complex hierarchical system, which our team eventually came to. We have one common big root and some small roots. Backend, mobile development, frontend, Yandex.Food - they all live each in their own separate repository. At the same time, information about all these projects is stored in a large common repository - in the root.
After you finally connect this versioning, install it with all your colleagues, where who will live and how, who will be engaged in support - after all this, you have to make a difficult choice.
TeamCity supports only two formats of configs. With XML, I suspect no one will want to work, so we chose the second format. It allows you to make these configs on a Kotlin script.
eamCity creates a maven-project, some semblance of any ordinary project. You can do one of two things with it: either upload to your project - Android, backend, it doesn’t matter - or leave it as a standalone project. Then you will have an independent repository with an independent project.
What is the plus of this approach? Personally, I and those guys who deal with our infrastructure on the backend and frontend, immediately bribed something. And even those who are not familiar with Kotlin, who heard about it for the first time, went and began to teach him.
These two lines create the whole project. This is the dialect of the TeamCity API. The API changes every major version. There are 2018-2, 2018-1, 2017, etc. Soon, hopefully, the 2019th will be released.
The second line simply declares the project.
Here is the project itself. This is absolutely real code. That's what our root repository looks like now. Nothing extra, nothing complicated. The only manual work that is required here is to manually create the UUID yourself. TeamCity requires that each object, each project has its own unique identifier. You can write anything there. I just use the standard nickname uuidgen team.
Here begins the adventure in the Kotlin DSL. I think this is a completely uncomplicated language for mastering. By uploading it to IDEA, Eclipse or any other IDE, you can get all the documentation, highlighting, autocomplete, tips. In fact, many of them are missing in the interface. Therefore, my personal experience says that working with the code is much more convenient, simpler and more intuitive. We are still developers.
Something like this looks like a real config now working at the same time, supporting TeamCity configs themselves. That is, TeamCity builds its own configs in its own environment. If all is well and everything has gone wrong, he calmly ships it in memory and replicates the changes to PostgreSQL. The base is already connected to the service itself. And here it will be a sin not to use all the features of Kotlin.
In this case, unlike XML, these configs can be described using polymorphism, inheritance - any features of the Kotlin language are allowed. The only important point is that all this can eventually turn into chaos that existed with us before we introduced versioning of the configs on the Kotlin script.
But, oddly enough, this chaos has become much less. Because before it was not quite obvious how to do what I want, how to achieve this or that feature? From the code, in my practice, it’s much easier to understand how to implement any feature.
The most interesting adventures begin here: how do we implement some things and how, in principle, make the interaction of the project with TeamCity easier?
Everyone here present in one form or another is preparing a release, participating in its assembly, in the publication. We publish our releases on various channels on Google Play.
We have beta, there are experiments, there is stable. We use a special plugin with a robot that posts comments with a report on the release build in the release ticket. And all this is set up with such a beautiful window. It appears as soon as you try to collect the release. These questions cannot be avoided.
From the TeamCity interface, it looks something like this. To immediately understand what, where, where and how, you need to read through each parameter, you need to experiment. From the documentation, in addition to what is visible on the screen, nothing else can be gleaned.
In code, it looks like this. At least so far, for half a year, no one has yet come and asked - how can I make some feature? Most often from the code it is all intuitively clear.
At the same time, some things are done quite simply, but hidden behind several layers of the interface. I have to walk, go back and forth.
Here is an example of how security is implemented in TeamCity. In my practice, for most people, TeamCity seems to be a fairly simple cold system that out of the box does not support integration, for example, with security services. Therefore, all tokens, all keys, all credentials with us most often stuck out in the open. Why not?
In fact, TeamCity is safe. He can create his own special file on his server, which is called - credential json, as shown. And he creates such a key for each token, for each credential, which we specially generate through the interface. You can already put it into the code and be sure that this credential will never come up in TeamCity logs or in the TeamCity interface. The system can cut these keys literally from anywhere. The whole interface is, roughly speaking, a kind of decoration.
Okay, we’ve set up some of our parameters, made a forward of the required parameters, for example, to build the release. But what if we want to go further? And we wanted to go further. During assembly, a lot of different steps are launched. We run several nested libraries that build from completely different repositories. And we just wanted to pull up fresh changes. All that is now. Do not bother - for example, do not collect an auxiliary library for pool requests, do not upload it to the maven repository, do not add additional gestures to the pool request.
We just set up the chain assembly. I will show to the end how unobvious and inconvenient it is to do this from the interface, in my personal opinion. And there already judge for yourself.
The chain assembly in the interface looks something like this.
It looks something like this in code. We simply indicate which particular configuration is dependent and what to do if any of the configurations did not work or was canceled by the user from the outside. In this case, I do not want the assembly to start at all. Because what is the point of it if we have not collected all the dependent libraries?
Around the same spirit, all other things are being done. And the entire project in TeamCity takes literally 500 lines of code.
It turns out that you can forward some interesting parameter through all the dependencies. I showed chaining for a reason. Chains is convenient, but hard to prepare in the interface. And TeamCity does not document such an important feature as forwarding through parameters. What is it for? Suppose, in our assembly in Gradle or somewhere else, we want to get tied to some specific field, forward the same address to the release ticket. And we want to do this once, and not for each nested assembly.
TeamCity has a not-so-obvious and completely non-documented parameter - reverse.dep (reverse dependency). It throws all of the parameters that come after the asterisk into all nested builds.
At the output, we get about such a simple structure. You can complicate it and make nesting as deep as your imagination or needs. And to be sure that in all these dependencies, in all these configurations, all of our parameters that we expect at each step of the assembly will be forwarded. Ready to answer your questions. Thanks to all!
All Articles