Monitoring
Why Prometheus
Prometheus has become the mainstream open source monitoring tool of choice in the container and microservice world.
It provides ready-to-use Exporters for all services that our system provides and implements a pull mechanism that is better suited for microservice architecture.
Furthermore, it supports additional features like the Alert Manager, a query language to retrieve the data and the visualization tool Grafana
Kubernetes Prometheus Stack
We are using the Kube-Prometheus Stack developed by the Prometheus community.
Installing the helm chart
Make sure your kubectx is set to the Airy Core instance and you have Helm installed.
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/kube-prometheus-stack
Customizing Prometheus
In this section you can customize the Prometheus chart by changing the defaults
in infrastructure/tools/prometheus/values.yaml to ones that suite your
requirements.
To access Prometheus, Grafana and Alertmanager from outside the cluster you have
to put your hostname in the respective hosts: [] variable.
In the case that you make Grafana publicly accessible you should also set the
adminPassword to something secure.
You can apply those changes by running:
helm upgrade prometheus --values infrastructure/tools/prometheus/values.yaml
Grafana Dashboards
Grafana is a very powerful visualization tool which can be used for all sorts of dashboarding and monitoring tasks.
For Grafana there is one more step to do before you can access it.
k edit cm prometheus-grafana
[server]
domain = <your_hostname>
root_url = <your_hostname>/grafana
serve_from_sub_path = true
Access Grafana under /grafana and login with the adminPassword you set in
the values.yaml.
Access Predefined Dashboards
If the defaultDashboardsEnabled is set to true you can find the default
Kubernetes dashboards under /grafana/dashboards
The Grafana website provides a lot more dashboards that can be added to your instance by importing them.
Here is a list of dashboard we recommend to add to monitor your Airy Core instance
Receiving alerts
To get notifications for the default alerts all you have to do is set up a receiver like described here. That way you can get notified on Slack or PagerDuty on issues like crashing components and nodes running out of free storage space.