Observability

Trilio Metrics Exporter for Prometheus
Trilio Metrics Exporter is a component of TrilioVault for Kubernetes written in Golang to export TrilioVault metrics. The project is based on the official Prometheus client library: prometheus_client.  
Trilio Metrics Exporter runs as a separate deployment and is installed by the Operator as part of the installation process. A complete list of Prometheus metrics and their details are provided in the Appendix section - TrilioVault Prometheus Metrics​
Consuming Metrics - Inside the Cluster
There are two methods to scrape the metrics from Trilio Exporter:
Using Annotations 
Prometheus can discover any metric in cluster exporter using the standard annotation set on the exporter pod.
The annotations are described below:
prometheus.io/scrape: The default configuration will scrape all pods and, if set to false, this annotation will exclude the pod from the scraping process.
prometheus.io/path: If the metrics path is not /metrics, define it with this annotation.
prometheus.io/port: Scrape the pod on the indicated port instead of the pod’s declared ports, Here it is 8080.
TrilioVault for Kubernetes has pre-configured the above annotations on the Metrics Exporter pod.  If you have a cluster with Prometheus already installed and configured, then your Prometheus instance should start collecting TrilioVault for Kubernetes metrics without any configuration change.
Using Scrape Job
You will need to configure a prometheus server to scrape the metrics from your newly running exporter. Add the following scrape job to your prometheus.yml configuration file.
1
  - job_name: trilio_exporter
2
    scrape_interval: 30s
3
    file_sd_configs:
4
    static_configs:
5
      - targets: ['EXPORTER_ADDRESS:8080']
Copied!
 EXPORTER_ADDRESS is the IP of the Prometheus exporter pod which can be found by running the following
1
#kubectl get pod -o wide |grep -i export
2
k8s-triliovault-exporter-787c7dd446-bcgkr            1/1   Running   0     6d4h  192.168.140.126  slave2  <none>      <none>
Copied!
​
Prometheus Operator Helm Chart Setup
If you installed prometheus  via helm as shown below helm install prometheus stable/prometheus-operator
Step 1: Expose TrilioVault Exporter: To expose the triliovault metrics, you need to create kubernetes Service. If your prometheus setup is not present in this cluster then expose via LoadBalancer Service.
You can create LoadBalancer Service using yaml
1
kind: Service
2
apiVersion: v1
3
metadata:
4
  name: k8s-triliovault-exporter-service
5
spec:
6
  ports:
7
    - name: web
8
      protocol: TCP
9
      port: 8080
10
      targetPort: 8080
11
  selector:
12
    app: k8s-triliovault-exporter
13
  type: NodePort
Copied!
 Step 2. Create ServiceMonitor: You will need to configure a ServiceMonitor for prometheus to scrape the metrics from your k8s-triliovault-exporter-service. Apply the following yaml to TVK (NS where TVK is installed)namespace.
1
  apiVersion: monitoring.coreos.com/v1
2
  kind: ServiceMonitor
3
  metadata:
4
    name: k8s-triliovault-exporter
5
    labels:
6
      app: k8s-triliovault-exporter
7
  spec:
8
    selector:
9
      matchLabels:
10
        app: k8s-triliovault-exporter
11
    endpoints:
12
    - port: web
Copied!
 Once the configuration is completed. You can find k8s-triliovault-exporter as active targets in Prometheus UI.
Please wait a couple of minutes for Prometheus to start scraping metrics. Access prometheus console at http://<prometheus-server-ip>/graph
TrilioVault Prometheus Metrics
If you leverage the Prometheus Operator stack, then configure the ServiceMonitor the same way as above configurations.
Consuming Metrics - Outside the cluster
To expose TVK metrics outside the cluster, perform the following two steps.
1.
Expose TrilioVault Exporter: TrilioVault metrics can be exposed outside the cluster via a LoadBalancer Service.

Leverage kubectl to create service for k8s-triliovault-exporter
1
kubectl expose deployment k8s-triliovault-exporter --type=LoadBalancer --name=exporter --namespace triliovault-integration
Copied!
OR you can create a LoadBalancer Service using a .yaml file
1
kind: Service
2
apiVersion: v1
3
metadata:
4
  name: k8s-triliovault-exporter-service
5
spec:
6
  ports:
7
    - protocol: TCP
8
      port: 8080
9
      targetPort: 8080
10
  selector:
11
    app: k8s-triliovault-exporter
12
    release: trilio
13
  type: LoadBalancer
Copied!
Once the Service is available capture the Public IP for k8s-triliovault-exporter-service, this will be used in second step.

2.
Scrape Job: You will need to configure a prometheus server to scrape the metrics from your newly running exporter. Add the following scrape job to your promethus.yml configuration file.
1
  - job_name: trilio_exporter
2
    scrape_interval: 30s
3
    file_sd_configs:
4
    static_configs:
5
      - targets: ['EXPORTER_PUBLIC_IP:8080']
Copied!
Visualizing Metrics with Grafana
Metrics from Prometheus can be visualized leveraging Grafana. Dashboards can be created in Grafana with the TrilioVault metrics exposed through Prometheus. 
TrilioVault Grafana Dashboards
Trilio provides pre-created Grafana dashboards with the product to make monitoring and observing your backup landscape easy. 
Grafana dashboards are pivoted on the following themes and provide high-level overview, summary and details around each theme.
1.
Backups
2.
Restores
3.
Targets
4.
BackupPlans/Application
Import Grafana Dashboards
There are two sources where TVK dashboards can be found
1.
TVK Grafana dashboards can be found on GitHub.
1.
These dashboards can be imported into a Grafana instance following instructions from the Grafana project page and using the 

2.
TVK dashboards can also be found on the Grafana within the Trilio org page  
1.
Instructions for downloading and importing are provided with the dashboards.
Grafana Dashboard Example

Installation and Operation - Previous

Operate

Next - Installation and Operation

Upgrade

Last modified 1yr ago

Copy link

Contents

Trilio Metrics Exporter for Prometheus

Consuming Metrics - Inside the Cluster

Prometheus Operator Helm Chart Setup

Consuming Metrics - Outside the cluster

Visualizing Metrics with Grafana