Disclaimer . Kostis Kapelonis - Developer advocate Codefresh, the first CI / CD platform for Kubernetes and containers, to defend and uphold the principles of software development. Codefresh's mission “Automate and simplify everything from code to the cloud.” As a software engineer, Kostis has many years of experience in containerizing applications, building CI / CD pipelines, and developing Java applications. He lives in Greece and loves roller skating.
As the proverb says, “if something causes pain, do it more often.” Continuous integration is, in essence, a repetition of the integration step with a high frequency in order to alleviate the “pain” it causes. This article is about this - about the “pain” of development and how to reduce it.
There is a lot of information about continuous integration (CI) and continuous delivery (CD). Blog posts use technical terms to explain what CI / CD methodologies mean, what they do and how they can help your company. Unfortunately, often both of these methodologies are associated with specific tools or even software providers. A typical conversation on this topic in a company is:
- Do you use continuous integration in your team?
- Yes, of course, we use tool X!
Let me tell you a little secret. Continuous integration and delivery are two approaches to code development that are completely unrelated to a particular tool or vendor. Despite the fact that there are tools and solutions that can help you in both cases (for example, Codefresh), in reality the company can practice CI / CD using only bash scripts and Perl single-line scripts. This is not very practical, but certainly quite possible.
Therefore, instead of falling into the general trap of explaining the essence of CI / CD using tools and technical terms, we will explain what these methodologies are based on the most important factor in the development process - people!
The Story of People: The Hard Times of Software Integration
Meet Alice, Bob, Charlie, David, and Elizabeth - all of them work for SoftwareCo Inc. over creating the SuperBigProject application. Alice, Bob, and Charlie are developers. David is a test engineer, and Elizabeth is a team project manager.
The traditional way to develop an application is as follows: Alice, Bob, and Charlie work on three different functions on their workstation. Each developer writes and tests the code individually. They use long branches of functions that exist for several weeks or even months before being combined into a final product.
At some point, project manager Elizabeth gathers the whole team and says: “People, we already need to release a release, so you try!”
After that, Alice, Bob and Charlie try to integrate all three of their functions into one branch. This is a very stressful time because these features have never been tested together. Many errors and problems appear out of nowhere, due to incorrect assumptions or problems with the environment, because, if you remember, up to this point all functions have been simply tested on different workstations, isolated from each other.
As soon as the “integration fever” is over, the combined result is passed on to David, who will perform additional manual and automatic testing. This period also takes a lot of time, because it is David who can approve or block the release, depending on how many critical errors are found. Everyone is closely watching David while he does his part, as his testing can reveal serious problems that could delay the release of the product.
Finally, the testing is completed, and Elizabeth joyfully announces that the software product is ready for packaging and shipping.
So, how do people feel in this imaginary, but very realistic story?
- Developers Alice, Bob, and Charlie are unhappy because they always find out about integration issues right before release. The integration period is similar to a shootout in which bullets arrive simultaneously from all sides.
- The tester David is constantly nervous because of the “twitching” nature of his work - there are calm periods when he just waits until the developers finish work on the functions, and there is a testing phase when he is just overwhelmed with work and must deal with unexpected test scripts, to in addition, at this time, the development team literally stands behind him.
- Elizabeth, as a project manager, is also unhappy. The integration phase is a critical link in the project. This is a busy period, as any unexpected problem pushes the release of the product. Elizabeth dreams of a software release without any surprises, but in practice this never happens. The “hit” of the integration stage in the project’s planned timeline is always a guessing game.
In general, all team members are unhappy. By the way, if your company is still developing such software, try to understand that this development process is detrimental to the morale of your team.
So, here the only and main problem is the integration phase, which occurs with every release of the software product. This is a pain point in the workflow that does not allow a team of programmers to free themselves from the stressful situation associated with the release of the program.
Adding Continuity to Integration
Now that we have seen what “integration” means, it is very easy to understand what “continuous integration” means. As the proverb says, “if something causes pain, do it more often.” Continuous integration is, in essence, a repetition of the integration step with a high frequency in order to alleviate the “pain” it causes.
The most obvious way to make the process less painful is to perform the integration more often after each merging of functions, rather than waiting for the official release.
When a team practices continuous integration, then:
- All objects are merged directly into the main branch.
- Developers work together, not in isolation. All functions are developed from the main line.
- A function is considered developed if mainline is fully operational and works on any machine, and not just on a separate workstation.
- Testing takes place automatically both at the level of individual elements or code objects, and at the mainline level.
This is the essence of continuous integration. Of course, there are other nuances of this methodology, there is a whole book on this topic, but the main thing is that instead of a single stressful period of integration, when everything is merged and tested at the same time, integration occurs all the time in a continuous way.
Continuous integration in the software development process is superior to conventional integration because:
- Reduces the number of surprises that appear when merging code objects.
- It solves the problem "the program only works on my machine."
- It splits the testing period into several periods, where each code object gradually merges with the main line, instead of merging all objects at once.
As a result, a team using CI does not feel like a roller coaster when quiet periods of development are interspersed with stressful releases. In addition, she gets the opportunity to soberly assess how close the project is to completion, instead of guessing the release dates. Using CI is one of the pillars of modern software development. Today this method is very well documented and well known. Therefore, there can be no justification for the fact that your company still does not practice CI when developing software projects.
Hard Times Software Delivery
Now that we’ve looked at the history of regular integration and how continuous integration works, we can take this to the next level of the development process — continuous delivery. If we return to our original story, we will see a picture similar to the process of normal integration:
Product launch was essentially an event akin to the Big Bang. After the software was considered tested, someone had to complete the process of folding and deploying software from containers. The deployment of software on production was also a very busy period and traditionally included many manual steps and follow-up procedures. Deployments were very rare, and even today there are companies that perform this procedure no more than once every six months. Only in extreme cases, the deployment took place at a time - using the cascading model of software development (waterfall).
Software delivery only after reaching the deadline creates the same problems as regular, infrequent integrations:
- The production environment is usually different from the testing environment, which requires additional configuration at the last minute.
- Functions that worked normally in a test environment break in the working environment.
- Functions that were not ready at the time of release, are either never provided to customers at all, or their refinement pushes the release even further.
- In this regard, releases create tension between developers who want to add new functions to expand the functionality of the software, and operators who want stability and do not want to deploy too many new functions at once.
The solution to this problem is the same template as in the case of integration. If we can alleviate the pain of the integration process by making it more frequent, then we can do the same in the software delivery process.
Add continuity to delivery
Continuous delivery is the practice of packaging in containers and preparing software as if it were being sent to production as often as possible. And the most extreme delivery method is after each merge.
In this way, the CD takes CI one step further. After merging each function with the mainline branch, the application is not only checked for correctness, but it is also packaged and deployed into a test environment that perfectly matches the working environment. All this happens in a fully automatic mode - pay attention to the absence in the figure of men who indicate manual procedures.
Also bear in mind that each new feature is a potential candidate for launch into production. In practice, not all candidates are sent for production. Depending on the organization of the process, the decision to deploy to production sometimes requires the intervention of a responsible person who decides whether to release the release into production or not, but does not personally participate in the preparation of the release. At this point, the release is already packaged, tested and deployed in a test environment.
Continuous delivery (CD) is a little more complicated than continuous integration (CI). The reason is that since each release candidate can potentially achieve production, the full life cycle should be automated:
- Assemblies must be repeatable and deterministic.
- All steps must be automated, which is quite difficult to put into practice.
- All configuration and related files must exist in the version control system, and not just in the source code.
- Each function / release should be tested in its own dynamically created and dynamically destroyed test environment.
- All test suites must be automated and relatively fast, which is also not an easy task.
Although a “cloud” can certainly help to fulfill all these requirements, a team of programmers, both developers and operators, needs a certain level of discipline that really ensures the process of continuous software delivery.
After the introduction of CD releases become a routine, as if carried out with a simple click of a button. Each team member, not just the project manager, can see the current release candidate. This candidate may not have some required functions or it may not fully satisfy all the requirements, but this is absolutely not important until it affects the release process itself.
The important fact is that the release is fully tested, packaged and ready to be sent to production if necessary. At the same time, any project participant should be able to give a green light and immediately release software for production.
If you use a CD, the software life cycle can be represented as follows:
Each release candidate is always prepared in advance. The person decides whether the release candidate will also be nominated for production. Release candidates who do not reach production continue to be stored as artifacts in case they need to be used in the future.
For your information, about continuous delivery, as well as about continuous integration, a whole book has also been written from which you can find out the details of this methodology.
Bonus: Continuous Deployment
The letter “D” in the CD may mean Deployment deployment, not Delivery delivery. This approach to the development process is based on continuous delivery and essentially eliminates human intervention. Any candidate for the release, who will pass all the tests and quality checks and will be recognized as ready, is immediately sent to production.
Admittedly, only a very small number of companies are able to operate in this way. Promotion to production without human involvement should not be taken lightly, since at the time of this writing, many companies do not even practice continuous delivery, not to mention continuous deployment.
You should understand that each subsequent approach to the development process is based on the implementation of the previous approach.
Before moving up the path of improvement, your company must make sure that each of the foundations of the process is truly solid. We at Codefresh saw a lot of companies that intended to switch to cloud technologies, trying to re-equip their technologies optimized for use in data centers on CI / CD pipelines, without realizing that some of these technologies are outdated today. Attempting to deploy continuous deployment without fully embracing continuous delivery is doomed to failure.
The following figure shows which stages of the software development and implementation process carried out by Alice, Bob, Charlie, David and Elizabeth cover CD and CI.
Make sure you approach each software development paradigm in the correct order. Believe that the introduction of continuous delivery is a very realistic goal, for the implementation of which there are all the necessary tools.
A bit of advertising :)
Thank you for staying with us. Do you like our articles? Want to see more interesting materials? Support us by placing an order or recommending it to your friends, cloud VPS for developers from $ 4.99 , a 30% discount for Habr users on the unique entry-level server analog that we invented for you: The whole truth about VPS (KVM) E5-2650 v4 (6 Cores) 10GB DDR4 240GB SSD 1Gbps from $ 20 or how to share a server? (options are available with RAID1 and RAID10, up to 24 cores and up to 40GB DDR4).
Dell R730xd 2 times cheaper? Only we have 2 x Intel TetraDeca-Core Xeon 2x E5-2697v3 2.6GHz 14C 64GB DDR4 4x960GB SSD 1Gbps 100 TV from $ 199 in the Netherlands! Dell R420 - 2x E5-2430 2.2Ghz 6C 128GB DDR3 2x960GB SSD 1Gbps 100TB - from $ 99! Read about How to Build Infrastructure Bldg. class c using Dell R730xd E5-2650 v4 servers costing 9,000 euros for a penny?