MONITORING IN CONTAINER ORCHESTRATION ARJON BUJUPI SENIOR DEVOPS

MONITORING IN CONTAINER ORCHESTRATION

ARJON BUJUPI SENIOR DEVOPS ENGINEER A@BUJUPI. ME

TOPICS MONITORING IN TRADITIONAL INFRASTRUCTURE VS CONTAINER ORCHESTRATION KUBERNETES OPERATORS PROMETHEUS OPERATOR DEPLOYING MONITORING MANIFESTS (DISCUSSION)

TRADITIONAL INFRASTRUCTURE Push based Most of the time you have to write custom monitoring scripts Map the parameters of the script with the parameters of the check Deploy these checks using a monitoring agent or run them via SSH (SSH into the node, run a particular command, get the output and parse it into a known format by the monitoring platform) Not flexible because of the design model Not applicable on modern infrastructure Very hard to develop

CONTAINER ORCHESTRATION PLATFORMS Pull based Monitoring platform gathers metrics and fires alerts based on the rules for these metrics We don’t monitor the application itself but the object running them (pods) The application itself is monitored by the platform (health checks) Flexible because the infrastructure is always changing No load on the infrastructure

KUBERNETES OPERATORS An Operator is a method of packaging, deploying and managing a Kubernetes application. A Kubernetes application is an application that is both deployed on Kubernetes, managed using the Kubernetes APIs and kubectl tooling. Usually the operator creates custom APIs and uses them to configure the application. It also manages the lifecycle of the application. Makes your life easier in case you have to manage the same application for different purposes (Saa. S, monitoring rules, credentials and more).

PROMETHEUS Prometheus is an open-source software application used for event monitoring and alerting. Records real-time metrics in a time series database built using a pull model, with flexible queries and real-time alerting. Harvests metrics from exporters (simple HTTP server exposing a txt based page with metrics). Powerful querying language allows you to define one alert rule for unlimited objects. Built-in visualizing. No load on the monitored object itself since it’s pull based.

PROMETHEUS OPERATOR The Prometheus Operator for Kubernetes provides easy monitoring definitions for Kubernetes services and deployment and management of Prometheus and Alertmanager instances. Easily launch a Prometheus instance for your Kubernetes namespace, a specific application or team easily using the Operator. Configure the fundamentals of Prometheus like versions, persistence, retention policies, and replicas from a native Kubernetes resource. Automatically generate monitoring target configurations based on familiar Kubernetes label queries; no need to learn a Prometheus specific configuration language.

CUSTOM RESOURCE DEFINITIONS Kind: Prometheus, which defines a desired Prometheus deployment. The Operator ensures at all times that a deployment matching the resource definition is running. Kind: Service. Monitor, which declaratively specifies how groups of services should be monitored. The Operator automatically generates Prometheus scrape configuration based on the definition. Kind: Prometheus. Rule, which defines a desired Prometheus rule file, which can be loaded by a Prometheus instance containing Prometheus alerting and recording rules. Kind: Alertmanager, which defines a desired Alertmanager deployment. The Operator ensures at all times that a deployment matching the resource definition is running.

EXAMPLE: MONITORING KAFKA 1. Deploy the Kafka exporter 2. In case Kafka is external, we have to create a Kubernetes endpoint and service for it 3. Create the service to expose metrics 4. Create a Service. Monitor to monitor this service 5. Create the Prometheus. Rule and apply alert rules based on your requirements

RECOMMENDED MONITORING SETUP Prometheus Operator 2 Prometheus instances running as replicas (not really a cluster) 2 Alertmanager instances running as replicas (not really a cluster) Node exporter running as a daemonset Grafana running as a stateless deployment (dashboards and datasources managed by configmaps) Kube state metrics (generates metrics about the state of the objects) Prometheus Adapter (for custom application metrics)

DEPLOYING MONITORING MANIFESTS Official Helm chart (not recommended) A configuration management tool (e. g. Ansible, good idea but requires maintenance) Kube-Prometheus by Core. OS (steep learning curve but very configurable)

QUESTIONS?

THANK YOU