> For the complete documentation index, see [llms.txt](https://docs.trilio.io/kubernetes/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.trilio.io/kubernetes/4.0.x/advanced-configuration/observability/logging-monitoring-and-measuring.md). # Observability of Trilio with Prometheus and Grafana ## About the Trilio Metrics Exporter for Prometheus The Trilio Metrics Exporter is installed as a component of Trilio for Kubernetes. It is used to export Trilio performance metrics in the Prometheus format, consumable by your existing Prometheus monitoring. The exporter is based on the [official Prometheus client library](https://github.com/prometheus/client_golang). For details on the Prometheus formatted metrics provided by the Trilio Metrics Exporter, check the [Exported Prometheus Metrics ](/kubernetes/4.0.x/advanced-configuration/observability/triliovault-prometheus-metrics.md)page. In this How-to, we'll show you how you can consume performance metrics from the Trilio Metrics Exporter with your existing Prometheus installation. {% hint style="info" %} In order to leverage the OpenShift built-in Prometheus instance hosted in the `openshift-monitoring` namespace, extra permissions will be required for the service account (`openshift-monitoring:prometheus-k8s`) to pull metrics from the openshift-operators namespace, where Trilio is deployed. {% endhint %} ## Consuming Metrics - Inside the Cluster There are two methods to scrape the metrics from Trilio Exporter: ### **Using Annotations** Prometheus can discover any metric in the cluster exporter using the standard annotation set on the exporter pod. The annotations are described below: * `prometheus.io/scrape:` The default configuration will scrape all pods; if set to false, this annotation will exclude the pod from the scraping process. * `prometheus.io/path:` If the metrics path is not /metrics, define it with this annotation. * `prometheus.io/port:` Scrape the pod on the indicated port instead of the pod’s declared ports; Here, it is 8080. {% hint style="success" %} **Trilio for Kubernetes has pre-configured the above annotations on the Metrics Exporter pod. If you have a cluster with Prometheus already installed and configured, then your Prometheus instance should start collecting Trilio for Kubernetes metrics without any configuration change.** {% endhint %} ### **Using Scrape Job** You will need to configure a Prometheus server to scrape the metrics from your newly running exporter. Add the following scrape job to your `prometheus.yml` configuration file. ``` - job_name: trilio_exporter scrape_interval: 30s file_sd_configs: static_configs: - targets: ['EXPORTER_ADDRESS:8080'] ``` `EXPORTER_ADDRESS` is the IP of the Prometheus exporter pod, which can be found by running the following ``` #kubectl get pod -o wide |grep -i export k8s-triliovault-exporter-787c7dd446-bcgkr 1/1 Running 0 6d4h 192.168.140.126 slave2 ``` ### Using ServiceMonitor **Step 1: Expose Trilio Exporter**: To expose the Trilio metrics, you need to create a Kubernetes Service. If your Prometheus setup is not present in this cluster, expose it via a LoadBalancer Service. You can create a LoadBalancer Service using the following.YAML ```yaml kind: Service apiVersion: v1 metadata: name: k8s-triliovault-exporter labels: app: k8s-triliovault-exporter spec: ports: - name: web protocol: TCP port: 8080 targetPort: 8080 selector: app: k8s-triliovault-exporter type: NodePort ``` **Step 2. Create ServiceMonitor**: You will need to configure a ServiceMonitor for Prometheus to scrape the metrics from your k8s-triliovault-exporter-service. Apply the following YAML to the namespace where T4K is installed. ```yaml apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: k8s-triliovault-exporter labels: app: k8s-triliovault-exporter spec: selector: matchLabels: app: k8s-triliovault-exporter endpoints: - port: web ``` Once the configuration is completed. You can find `k8s-triliovault-exporter` active targets in Prometheus UI. Please wait a couple of minutes for Prometheus to start scraping metrics. Access Prometheus console at http\://**\**/graph ![Trilio Prometheus Metrics](/files/-M79PrzwYwKDI3RZyvBg) {% hint style="info" %} If you leverage the Prometheus Operator stack, configure the ServiceMonitor the same way as the above configurations. {% endhint %} ### Consuming Metrics - Outside the cluster To expose T4K metrics outside the cluster, perform the following two steps. 1. **Expose Trilio Exporter**: Trilio metrics can be exposed outside the cluster via a LoadBalancer Service.\ \ Leverage *kubectl* to create a service for k8s-triliovault-exporter ``` kubectl expose deployment k8s-triliovault-exporter --type=LoadBalancer --name=exporter --namespace triliovault-integration ``` OR you can create a LoadBalancer Service using a .yaml file ```yaml kind: Service apiVersion: v1 metadata: name: k8s-triliovault-exporter labels: app: k8s-triliovault-exporter spec: ports: - protocol: TCP port: 8080 targetPort: 8080 selector: app: k8s-triliovault-exporter type: LoadBalancer ``` \ Once the Service is available, capture the Public IP for k8s-triliovault-exporter-service, this will be used in the second step. 1. **Scrape Job**: You will need to configure a Prometheus server to scrape the metrics from your newly running exporter. Add the following scrape job to your `prometheus.yml` configuration file. ``` - job_name: trilio_exporter scrape_interval: 30s file_sd_configs: static_configs: - targets: ['EXPORTER_PUBLIC_IP:8080'] ``` ## Visualizing Metrics with Grafana Metrics from Prometheus can be visualized leveraging Grafana. Dashboards can be created in Grafana with the Trilio metrics exposed through Prometheus. #### Trilio Grafana Dashboards Trilio provides pre-created Grafana dashboards to make monitoring and observing your backup landscape accessible. Grafana dashboards are pivoted on the following themes and provide a high-level overview, summary, and details around each theme. 1. Backups 2. Restores 3. Targets 4. BackupPlans/Application #### Import Grafana Dashboards T4K dashboards can be found on the Grafana within the [Trilio org page.](https://grafana.com/orgs/triliovault) {% hint style="danger" %} Please ensure that the dependencies mentioned for each dashboard are fulfilled. The T4K dashboards will not work as expected if these dependencies are unmet. {% endhint %} 1. The dashboards can be imported into a Grafana instance following instructions from the Grafana [project page](https://grafana.com/docs/grafana/latest/reference/export_import/). 2. Instructions for downloading and importing are provided with the dashboards. ![Grafana Dashboard Example](/files/-MBpkAh81L1xuHEbxYO2)