How to run multiple pipelines using GitLab CI / CD

Launch and visualization of pipelines when configuring GitLab CI / CD for several projects.

Continuous Integration (CI) is the practice of automating the assembly and testing of code before it merges with the main branch. It allows developers to inject code quite often and early, while reducing the risk of introducing new errors into the main source code repository.

Although CI checks that the new code will not break when integrated with other code in the same repo, passing all the tests on this repo is only the first step. After running CI in code, it is important to deploy and run tests in a real environment. The transition from CI to Continuous Delivery and Deployment (CD) is the next step to the “adult” DevOps. Deployment and subsequent retesting allows you to test the code of one project along with other components and services that may be managed by other projects.

Why do I need to make sure my code works with other components?

A good example is the architecture of microservices. Typically, microservices are managed in different projects , where each microservice has its own repository with a pipeline. In addition, very often each development team is responsible for individual microservices and pipeline configurations. As a programmer, you might want to make sure that changes to your code do not violate the functionality of its microservices. Therefore, you can run tests on them in addition to tests for your project.

Pipeline Cross Project

When starting a project pipeline, you will also need to run cross-project pipeline, which will ultimately deploy and test the latest version of all dependent microservices. To achieve this, you need a simple, flexible and convenient way to launch other pipelines within the framework of the CI of your project. GitLab CI / CD offers an easy way to launch a cross-project pipeline by adding a special task to the CI configuration file.

GitLab CI / CD configuration file

In GitLab CI / CD, pipelines, as well as jobs and the steps of their components are defined in the .gitlab-ci.yml

for each project. The file is part of the project repository. It is fully versioned, and developers can edit it using any IDE of their choice. They do not need to ask the system administrator or the DevOps team to make changes to the pipeline configuration, as they can do it themselves. The .gitlab-ci.yml

determines the structure and order of the pipelines, and decides what needs to be done using the GitLab Runner (the agent that runs the tasks), and what decisions should be made when certain conditions arise, for example, when the process completes successfully or exits system.

Adding job to run cross-project pipeline

Starting with GitLab 11.8, GitLab provides a new CI / CD configuration syntax for running cross-project pipelines, which can be found in the pipeline configuration rules . The following code illustrates how to configure a bridge job to run a downlink pipeline:

 // job1  job    deploy: stage: Deploy script: this is my script // job2  bridge job   ,   -  Android: stage: Trigger-cross-projects trigger: mobile/android

In the above example, as soon as the deploy job at the deployment stage is successful, the task for Android bridge starts. Its initial status will be pending. GitLab will create a downward pipeline in the mobile / android project, and once it is created, the Android job will succeed. In this case, mobile / android is the full path to this project.

The user who created the upstream pipeline must have access rights to the downstream project (in this case, mobile / android). If the subordinate project cannot be found or the user does not have access rights to create a pipeline there, the Android job will receive the failed status.

Overview of charts from ascending to downstream pipeline

GitLab CI / CD allows you to visualize the pipeline configuration. In the figure below, the assembly, testing, and deployment steps are part of an upstream project. Once the deploy job is completed successfully, four cross-projects will be launched in parallel, and you can proceed to them by clicking on one of the downstream job.

In the figure below you can see the downstream pipeline “Service - Finance”. Now you can scroll left to the ascending pipeline, scroll right back to the descending pipeline, or select another descending pipeline.

Defining a child pipeline branch

You can specify the name of the branch that the descending pipeline will use:

 trigger: project: mobile/android branch: stable-11-2

Use the project keyword to indicate the full path to the downstream project. Use the branch keyword to determine the name of the branch. GitLab will use the commit that is currently in the HEAD branch when creating the downlink pipeline.

Passing variables to a descending pipeline

Someday, you might want to pass variables to a descending pipeline. You can do this with keywords for variables, as you would with a regular job definition.

 Android: variable: ENVIRONMENT: 'This is the variable value for the downstream pipeline' stage: Trigger-cross-projects trigger: mobile/android

The ENVIRONMENT variable will be passed to each job defined in the descending pipeline. It will be available as an environment variable each time GitLab Runner selects a job.

Total Cross-Project Pipeline

The .gitlab-ci.yml

determines the order of the CI / CD stages, which tasks to perform and under what conditions to start or skip the task. Adding a 'bridge job' with the trigger

keyword to this file can be used to run cross-project pipelines. We can pass parameters to jobs in descending pipelines and even define the branch that the descending pipeline will use.

Pipelines can be complex structures with many sequential and parallel tasks, and, as we just learned, sometimes they can run downstream pipelines. To make it easier to understand the pipeline flow, including downstream pipelines, GitLab has pipline graphics for viewing pipelines and their statuses.

Also read other articles on our blog:

All Articles