In short, GitOps is a set of practical methods for using Git pull requests to manage infrastructure and application configurations. Git repository within GitOps is considered as one single source of information about the state of the system, and any changes in this state are fully monitored and audited.
The idea of change tracking in GitOps is by no means new; this approach has long been used almost everywhere when working with application source code. GitOps simply implements similar functions (review checks, pull requests, tags, etc.) when managing infrastructure and application configurations and provides similar advantages as in the case of source code management.
For GitOps, there is no academic definition or approved set of rules, only a set of principles on which this practice is based:
- The declarative description of the system is stored in the Git repository (configs, monitoring, etc.).
- State changes are made through pull requests.
- The state of running systems is aligned with the data in the repository using Git push requests.
GitOps Principles
- System definitions are described as source code.
System configuration is considered as code, so it can be stored and automatically versioned in the Git repository, which serves as the only source of truth. This approach makes it easy to rollout and rollback changes to systems.
- The desired state and configuration of systems are set and versioned in Git
By storing and versioning in Git the desired state of the systems, we get the ability to easily roll and roll back changes to systems and applications. We can also use Git's security mechanisms to control code ownership and verify its authenticity.
- Configuration changes can be applied automatically using pull queries.
Using Git pull requests, we can easily control how changes are applied to configurations in the repository. For example, they can be sent to other team members for verification or run through CI tests, etc.
And at the same time, you do not need to give admin authority to the right and left. To commit configuration changes, users have sufficient permissions in the Git repository where these configurations are stored.
- Troubleshooting Uncontrolled Configuration Drift
When the desired state of the system is stored in the Git repository, we can only find software that would control that the current state of the system corresponds to its desired state. If this is not so, then this software should - depending on the settings - either fix the discrepancy on its own or notify us of the configuration drift.
GitOps models for OpenShift
On-Cluster Resource Reconciller
According to this model, the cluster has a controller that is responsible for comparing Kubernetes resources (YAML files) in the Git repository with real cluster resources. In case of discrepancies, the controller sends out notifications and, possibly, takes measures to eliminate inconsistencies. This GitOps model is used by Anthos Config Management and Weaveworks Flux.
External Resource Reconciler (Push)
This model can be considered as a variation of the previous one, when we have one or more controllers responsible for synchronizing resources in pairs “Git repository - Kubernetes cluster”. The difference here is that each managed cluster does not have to have its own separate controller. “Git - k8s cluster” pairs are often defined as CRD descriptions (custom resources definition), in which you can describe how the controller should perform synchronization. In this model, controllers compare the Git repository specified in CRD with the resources of the Kubernetes cluster, which are also defined in CRD, and perform the corresponding actions based on the comparison results. In particular, such a GitOps model is used in ArgoCD.
GitOps on the OpenShift Platform
Multicluster Kubernetes Infrastructure Administration
With the spread of Kubernetes and the growing popularity of multi-cloud strategies and edge computing, the average number of OpenShift clusters per customer has also increased.
For example, when using peripheral computing, clusters of one customer can be deployed in hundreds or even thousands. As a result, he is forced to manage several independent or coordinated OpenShift clusters in the public cloud and on-premise.
At the same time, a lot of problems have to be solved, in particular:
- To control that clusters are in an identical state (configs, monitoring, storage, etc.)
- Re-create (or restore) clusters according to a known state.
- Create new clusters according to a known state.
- Pull changes across multiple OpenShift clusters.
- Roll back changes to multiple OpenShift clusters.
- Link patterned configurations with different environments.
Application configurations
During their life cycle, applications often go through a chain of clusters (dev, stage, etc.) before they get into a production cluster. In addition, due to availability and scalability requirements, customers often deploy applications on multiple on-premise clusters or in multiple regions of a public cloud platform.
In this case, it is necessary to solve the following problems:
- Ensure the movement of applications (binaries, configs, etc.) between clusters (dev, stage, etc.).
- Pull changes in applications (binaries, configs, etc.) in several OpenShift clusters.
- Roll back changes in applications to the level of the previous known state.
OpenShift GitOps Usage Scenarios
1. Apply changes from the Git repository
The cluster administrator can store the OpenShift cluster configurations in the Git repository and automatically apply them to create new clusters without any extra effort and bring them into a state identical to the known state that is stored in the Git repository.
2. Sync with Secret Manager
The administrator will also find it useful to synchronize OpenShift secret objects with appropriate software like Vault in order to manage them using tools specially created for this.
3. Control drift configurations
The admin will only be in favor if OpenShift GitOps will detect and warn about discrepancies between real configurations and those specified in the repository, so that you can quickly respond to drift.
4. Configuration Drift Notifications
Useful in the case when the admin wants to quickly find out about drift configurations in order to quickly take appropriate measures on their own.
5. Manual synchronization of configurations when drifting
Allows the administrator to synchronize the OpenShift cluster with the Git repository in case of drift configurations in order to quickly return the cluster to a previous known state.
6. Auto-synchronization of drift configurations
The administrator can also configure the OpenShift cluster to automatically synchronize with the repository when a drift is detected, so that the cluster configuration always matches the configs in Git.
7. Multiple Clusters - One Repository
Admin can store configurations of several different OpenShift clusters in one Git repository and selectively apply them as needed.
8. Hierarchy of cluster configurations (inheritance)
The admin can set the hierarchy of cluster configurations in the repository (stage, prod, app portfolio, etc. with inheritance). In other words, it can determine how configurations should be applied - to one or several clusters.
For example, if the admin defines the hierarchy “Production clusters (prod) → Clusters of system X → Production clusters of system X” in the Git repository, then the following configs are applied to the production clusters of system X:
- Configs common to all production clusters.
- Configs for cluster system X.
- Configs for the production cluster of system X.
9. Patterns and configuration overrides
The administrator can override the set of inherited configs and their values, for example, to fine-tune the configuration for specific clusters to which they will be applied.
10. Selective include'y and exclude'y for configurations, application configuration
The administrator can set the conditions for applying or not applying certain configurations to clusters with certain characteristics.
11. Pattern support
Developers will find it useful to choose how application resources will be determined (Helm Chart, pure Kubernetes yaml, etc.) in order to use the most suitable format for each specific application.
GitOps tools on the OpenShift platform
Argocd
ArgoCD implements the External Resource Reconcile model and offers a centralized UI for orchestrating relationships between clusters and Git repositories in a one-to-many fashion. The disadvantages of this program include the inability to manage applications while ArgoCD is not working.
Official site
Flux
Flux implements the On-Cluster Resource Reconcile model, and as a result, there is no centralized management of the definition repository, which is a weak point. On the other hand, precisely because of the lack of centralization, the ability to manage applications is retained even if one cluster fails.
Official site
Install ArgoCD on OpenShift
ArgoCD offers an excellent command line interface and web console, so we will not consider Flux and other alternatives here.
To deploy ArgoCD on the OpenShift 4 platform, follow these steps as a cluster administrator:
Deploying ArgoCD components on the OpenShift platform
# Create a new namespace for ArgoCD components oc create namespace argocd # Apply the ArgoCD Install Manifest oc -n argocd apply -f https://raw.githubusercontent.com/argoproj/argo-cd/v1.2.2/manifests/install.yaml # Get the ArgoCD Server password ARGOCD_SERVER_PASSWORD=$(oc -n argocd get pod -l "app.kubernetes.io/name=argocd-server" -o jsonpath='{.items[*].metadata.name}')
Refining ArgoCD Server to be seen by OpenShift Route
# Patch ArgoCD Server so no TLS is configured on the server (--insecure) PATCH='{"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"argocd-server"}],"containers":[{"command":["argocd-server","--insecure","--staticassets","/shared/app"],"name":"argocd-server"}]}}}}' oc -n argocd patch deployment argocd-server -p $PATCH # Expose the ArgoCD Server using an Edge OpenShift Route so TLS is used for incoming connections oc -n argocd create route edge argocd-server --service=argocd-server --port=http --insecure-policy=Redirect
Deploy ArgoCD Cli Tool
# Download the argocd binary, place it under /usr/local/bin and give it execution permissions curl -L https://github.com/argoproj/argo-cd/releases/download/v1.2.2/argocd-linux-amd64 -o /usr/local/bin/argocd chmod +x /usr/local/bin/argocd
Change admin password ArgoCD Server
# Get ArgoCD Server Route Hostname ARGOCD_ROUTE=$(oc -n argocd get route argocd-server -o jsonpath='{.spec.host}') # Login with the current admin password argocd --insecure --grpc-web login ${ARGOCD_ROUTE}:443 --username admin --password ${ARGOCD_SERVER_PASSWORD} # Update admin's password argocd --insecure --grpc-web --server ${ARGOCD_ROUTE}:443 account update-password --current-password ${ARGOCD_SERVER_PASSWORD} --new-password
After completing these steps, you can work with ArgoCD Server through the ArgoCD WebUI web console or the ArgoCD Cli command-line tool.
https://blog.openshift.com/is-it-too-late-to-integrate-gitops/
GitOps - It's Never Too Late
“The train has left” - this is what they say about the situation when the opportunity to do something is missed. In the case of OpenShift, the desire to immediately start using this new cool platform often creates just such a situation with the management and maintenance of routes, deployments and other OpenShift objects. But is the chance always completely missed?
Continuing a series of articles about GitOps , today we will show how to transform a manually created application and its resources into a certain process where the GitOps toolkit controls everything. To do this, we first deploy the httpd application with our hands. The screenshot below shows how we create a namespace, deployment, and service, and then do expose for this service to create a route.
oc create -f https://raw.githubusercontent.com/openshift/federation-dev/master/labs/lab-4-assets/namespace.yaml oc create -f https://raw.githubusercontent.com/openshift/federation-dev/master/labs/lab-4-assets/deployment.yaml oc create -f https://raw.githubusercontent.com/openshift/federation-dev/master/labs/lab-4-assets/service.yaml oc expose svc/httpd -n simple-app
So, we have a manually created application. Now it must be transferred under the control of GitOps without loss of availability. In short, it does this:
- Create a Git repository for the code.
- We export our current objects and load them into the Git repository.
- Select and deploy the GitOps toolkit.
- Add our repository to this toolkit.
- Defining the application in our GitOps toolkit.
- Perform a trial run of the application using the GitOps toolkit.
- We synchronize objects using the GitOps toolkit.
- We turn on pruning and auto-synchronization of objects.
As mentioned in a previous article , GitOps has one and only one source of information about all objects in the Kubernetes cluster (s) - the Git repository. Further, we proceed from the premise that your organization already uses a Git repository. It can be public or private, but it must be available to Kubernetes clusters. This can be the same repository as for application code, or a separate repository created specifically for deployment. It is recommended that you have strict permissions in the repository, as secret objects, routes, and other security-sensitive things will be stored there.
In our example, we will create a new public repository on GitHub. You can name it whatever you like, we use the name blogpost.
If the YAML files of the objects were not stored locally or in Git, you will have to use the oc or kubectl binaries. In the screenshot below, we request YAML for our namespace, deployment, service, and route. Before that, we cloned the newly created repository and moved to it with the cd command.
oc get namespace simple-app -o yaml --export > namespace.yaml oc get deployment httpd -o yaml -n simple-app --export > deployment.yaml oc get service httpd -o yaml -n simple-app --export > service.yaml oc get route httpd -o yaml -n simple-app --export > route.yaml
Now fix the deployment.yaml file to remove a field from it that the Argo CD cannot sync.
sed -i '/\sgeneration: .*/d' deployment.yaml
In addition, you need to change the route. We will first set the multiline variable, and then replace ingress: null with the contents of this variable.
export ROUTE=" ingress:\\ - conditions:\\ - status: 'True'\\ type: Admitted" sed -i "s/ ingress: null/$ROUTE/g" route.yaml
So, we figured out the files, it remains to save them in the Git repository. After that, this repository becomes the only source of information, and any manual changes to objects must be strictly prohibited.
git commit -am 'initial commit of objects' git push origin master
Further, we assume that you already have ArgoCD deployed (how to do this, see the previous post ). Therefore, we add to the Argo CD the repository we created that contains the application code from our example. Just make sure that you specify the exact repository that you created earlier.
argocd repo add https://github.com/cooktheryan/blogpost
Now create the application. The application sets values so that the GitOps toolkit understands which repository and paths to use, which OpenShift is needed to manage objects, as well as which specific branch of the repository is needed, and whether resources should be auto-synchronized.
argocd app create --project default \ --name simple-app --repo https://github.com/cooktheryan/blogpost.git \ --path . --dest-server https://kubernetes.default.svc \ --dest-namespace simple-app --revision master --sync-policy none
After the application is specified in the Argo CD, this toolkit starts checking already deployed objects for compliance with the definitions in the repository. In our example, auto-sync and cleanup are disabled, so the elements do not change yet. Please note that in the Argo CD interface, our application will have the status “Out of Sync” (Not synchronized), since there is no label tag affixed by ArgoCD.
That is why, when we start synchronization a little later, the redeployment of objects will not be performed.
Now run a test run to make sure that there are no errors in our files.
argocd app sync simple-app --dry-run
If there are no errors, then you can proceed to synchronization.
argocd app sync simple-app
After running the argocd get command on our application, we should see that the status of the application has changed to Healthy or Synced. This will mean that all resources in the Git repository now correspond to those resources that are already deployed.
argocd app get simple-app Name: simple-app Project: default Server: https://kubernetes.default.svc Namespace: simple-app URL: https://argocd-server-route-argocd.apps.example.com/applications/simple-app Repo: https://github.com/cooktheryan/blogpost.git Target: master Path: . Sync Policy: <none> Sync Status: Synced to master (60e1678) Health Status: Healthy ...
But now you can turn on auto-synchronization and cleanup to ensure that nothing will be created manually and that each time the object is created or updated in the repository, the deployment will be performed.
argocd app set simple-app --sync-policy automated --auto-prune
So, we successfully transferred to the control of GitOps an application that initially did not use GitOps.