Part 0 - introductory: how did we get to this?
The idea to write our own plugin for Grafan was born by chance. Our company has been monitoring web projects of various levels of complexity for more than 10 years. During this time, we have accumulated a lot of expertise, interesting cases, and experience in using various monitoring systems. And at some point we asked ourselves: “Is there a magical tool for monitoring Kubernetes, so that, as they say,“ set and forget ””? .. Promstandart for monitoring k8s, of course, has long been a bunch of Prometheus + Grafana. And as ready-made solutions for this stack, there is a large set of various kinds of tools: prometheus-operator, dashboards set kubernetes-mixin, grafana-kubernetes-app.
The grafana-kubernetes-app plugin seemed to be the most interesting option for us, but it has not been supported for more than a year and, moreover, does not know how to work with new versions of node-exporter and kube-state-metrics. And at some point, we decided: "But do not we make our own decision?"
What ideas we decided to implement in our plugin:
- visualization of the "application map": convenient presentation of applications in the cluster, grouped by namespace, deployment ...;
- visualization of connections of the form "deployment - service (+ ports)".
- visualization of the distribution of cluster applications by cluster nods.
- collecting metrics and information from several sources: Prometheus and k8s api server.
- monitoring of both the infrastructure part (use of processor time, memory, disk subsystem, network), and application logic - health-status pods, the number of available replicas, information about the passage of liveness / readyness samples.
Part 1: What is the Grafana Plugin?
From a technical point of view, the plugin for Grafana is an angular controller that is stored in the Grafan data directory ( /var/grafana/plugins/<your_plugin_name>/dist/module.js ) and can be loaded as a SystemJS module. Also in this directory should be a plugin.json file containing all the meta-information about your plugin: name, version, type of plugin, links to the repository / site / license, dependencies, and so on.
module.ts
plugin.json
As you can see in the screenshot, we specified plugin.type = app. For plugins for Grafana can be of three types:
panel : the most common type of plug-in - it is a panel for visualizing any metrics, it is used to build various dashboards.
datasource : a plug-in connector to a data source (for example, Prometheus-datasource, ClickHouse-datasource, ElasticSearch-datasource).
app : a plugin that allows you to build your own frontend application inside Grafana, create your own html pages and manually access the datasource to visualize various data. Also, plugins of other types (datasource, panel) and various dashboards can be used as dependencies.
An example of plugin dependencies with type = app .
As a programming language, you can use both JavaScript and TypeScript (we opted for it). You can find the blanks for hello-world plug-ins of any type here : in this repository there are a large number of starter-packs (there is even an experimental example of a plugin on React) with pre-installed and configured builders.
Part 2: preparing your local environment
To work on the plugin, we naturally need a kubernetes cluster with all the preinstalled tools: prometheus, node-exporter, kube-state-metrics, grafana. The environment should set up quickly, easily and naturally, and to ensure hot-reload data, the Grafana directory should be mounted directly from the developer's machine.
The most convenient, in our opinion, way of working locally with kubernetes is minikube . The next step is to establish the Prometheus + Grafana bundle using the prometheus-operator. This article details the process of installing prometheus-operator on minikube. To enable persistence, you must set the persistence: true parameter in the charts / grafana / values.yaml file, add your own PV and PVC and specify them in the persistence.existingClaim parameter
The final minikube launch script looks like this:
minikube start --kubernetes-version=v1.13.4 --memory=4096 --bootstrapper=kubeadm --extra-config=scheduler.address=0.0.0.0 --extra-config=controller-manager.address=0.0.0.0 minikube mount /home/sergeisporyshev/Projects/Grafana:/var/grafana --gid=472 --uid=472 --9p-version=9p2000.L
Part 3: development itself
Object model
In preparation for the implementation of the plugin, we decided to describe all the basic Kubernetes entities that we will work with as TypeScript classes: pod, deployment, daemonset, statefulset, job, cronjob, service, node, namespace. Each of these classes inherits from the general BaseModel class, which describes the constructor, destructor, methods for updating and switching visibility. Each class describes nested relationships with other entities, for example, a list of pods for an entity of type deployment.
import {Pod} from "./pod"; import {Service} from "./service"; import {BaseModel} from './traits/baseModel'; export class Deployment extends BaseModel{ pods: Array<Pod>; services: Array<Service>; constructor(data: any){ super(data); this.pods = []; this.services = []; } }
Using getters and setters, we can display or set the metrics of entities we need in a convenient and readable way. For example, the formatted output of allocatable cpu nods:
get cpuAllocatableFormatted(){ let cpu = this.data.status.allocatable.cpu; if(cpu.indexOf('m') > -1){ cpu = parseInt(cpu)/1000; } return cpu; }
Pages
A list of all the pages of our plugin is initially described in our pluing.json in the dependency section:
In the block for each page, we must indicate the PAGE NAME (it will then be converted to slug, by which this page will be available); name of the component responsible for the operation of this page (the list of components is exported to module.ts); specifying the role of the user for whom access to this page is available and navigation settings for the sidebar.
In the component responsible for the operation of the page, we must install templateUrl, passing there the path to the html-file with markup. Inside the controller, through dependency injection, we can access up to 2 important angular services:
- backendSrv - a service that provides interaction with the grafana api-server;
- datasourceSrv - a service that provides local interaction with all datasource installed in your Grafana (for example, the .getAll () method - returns a list of all installed datasource'ov; .get (<name>) - returns an instance object of a particular datasource.
Part 4: datasource
From the point of view of Grafana, the datasource is exactly the same plug-in as everyone else: it has its own entry point module.js, there is a file with meta-information plugin.json. When developing a plugin with type = app, we can interact with existing datasource (for example, prometheus-datasource), as well as our own, which we can store directly in the plugin directory (dist / datasource / *) or set as a dependency. In our case, the datasource comes with the plugin code. It is also necessary to have the config.html template and the ConfigCtrl controller that will be used for the configuration page of the datasource instance and the Datasource controller, which implements the logic of your datasource.
In the KubeGraf plugin, from the point of view of the user interface, the datasource is an instance of the kubernetes cluster in which the following features are implemented (the source code is available by reference ):
- data collection from k8s api-server (getting a list of namespace'ov, deployment'ov ...)
- proxying requests in prometheus-datasource (which is selected in the plugin settings for each specific cluster) and formatting responses for using data in both static pages and dashboards.
- updating data on static pages of the plugin (with the refresh rate set).
- processing requests for generating a template list in grafana-dashboards (.metriFindQuery () method)
- test connection with the final k8s cluster.
testDatasource(){ let url = '/api/v1/namespaces'; let _url = this.url; if(this.accessViaToken) _url += '/__proxy'; _url += url; return this.backendSrv.datasourceRequest({ url: _url, method: "GET", headers: {"Content-Type": 'application/json'} }) .then(response => { if (response.status === 200) { return {status: "success", message: "Data source is OK", title: "Success"}; }else{ return {status: "error", message: "Data source is not OK", title: "Error"}; } }, error => { return {status: "error", message: "Data source is not OK", title: "Error"}; }) }
A separate interesting point, in our opinion, is the implementation of the authentication and authorization mechanism for datasource. As a rule, out of the box for configuring access to the final data source, we can use the built-in Grafana component - datasourceHttpSettings. Using this component, we can configure access to the http data source by specifying the url and basic authentication / authorization settings: login-password, or client-cert / client-key. In order to realize the ability to configure access using a bearer token (de facto the standard for k8s), I had to do it a bit.
To solve this problem, you can use Grafana's built-in “Plugin Routes” mechanism (more on the official documentation page ). In the settings of our datasource, we can declare a set of routing rules that will be processed by the grafana proxy server. For example, for each individual endpoint there is the possibility of affixing headers or urls with the ability to template, data for which can be taken from the jsonData and secureJsonData fields (for storing passwords or tokens in encrypted form). In our example, requests of the form / __ proxy / api / v1 / namespaces will be proxied to the url of the form
<your_k8s_api_url> / api / v1 / namespaces with the Authorization: Bearer header.
Naturally, to work with the k8s api-server, we need a user with readonly accesses, the manifest for creating which you can also find in the source code of the plugin .
Part 5: release
After you write your own plugin for Grafana, you will naturally want to put it in the public domain. Grafana is a plugin library available at grafana.com/grafana/plugins
In order for your plugin to be available in the official store, you need to do PR in this repository by adding the following contents to the repo.json file:
where version is the version of your plugin, url is a link to the repository, and commit is a hash of the commit, by which a specific version of the plugin will be available.
And at the exit you will see a wonderful picture of the form:
The data for it will be automatically grabbed from your Readme.md, Changelog.md and the plugin.json file with the plugin description.
Part 6: instead of conclusions
We did not stop developing our plugin after the release. And now we are working on the correct monitoring of the use of the resources of the cluster nodes, the introduction of new features to increase UX, as well as raking a large amount of feedback received after installing the plugin both by our clients and from the ishui on the github (if you leave your issue or pull request, I I will be very happy :-)).
We hope that this article will help you understand such a great tool as Grafana and, possibly, write your own plugin.
Thanks!)