LogoLogo
4.0.X
4.0.X
  • About Trilio for Kubernetes
    • Welcome to Trilio For Kubernetes
    • Version 4.0.X Release Highlights
    • Compatibility Matrix
    • Marketplace Support
    • Features
    • Use Cases
  • Getting Started
    • Getting Started with Trilio on Red Hat OpenShift (OCP)
    • Getting Started with Trilio for Upstream Kubernetes (K8S)
    • Getting Started with Trilio for AWS Elastic Kubernetes Service (EKS)
    • Getting Started with Trilio on Google Kubernetes Engine (GKE)
    • Getting Started with Trilio on VMware Tanzu Kubernetes Grid (TKG)
    • More Trilio Supported Kubernetes Distributions
      • General Installation Prerequisites
      • Rancher Deployments
      • Azure Cloud AKS
      • Digital Ocean Cloud
      • Mirantis Kubernetes Engine
      • IBM Cloud
    • Licensing
    • Using Trilio
      • Overview
      • Post-Install Configuration
      • Management Console
        • About the UI
        • Navigating the UI
          • UI Login
          • Cluster Management (Home)
          • Backup & Recovery
            • Namespaces
              • Namespaces - Actions
              • Namespaces - Bulk Actions
            • Applications
              • Applications - Actions
              • Applications - Bulk Actions
            • Backup Plans
              • Create Backup Plans
              • Backup Plans - Actions
            • Targets
              • Create New Target
              • Targets - Actions
          • Trilio Monitoring
          • Resource Management
          • Guided Tours
        • UI How-to Guides
          • Multi-Cluster Management
          • Creating Backups
            • Cleanup Failed Backups
          • Restoring Backups
            • Cross-Cluster Restores
          • Monitoring Details
          • Disaster Recovery Plan
          • Continuous Restore
      • Command-Line Interface
        • YAML Examples
        • Trilio Helm Operator Values
    • Upgrade
    • Air-Gapped Installations
    • Uninstall
  • Reference Guides
    • T4K Pod/Job Capabilities
      • Resource Quotas
    • Trilio Operator API Specifications
    • Custom Resource Definition - Application
  • Advanced Configuration
    • AWS S3 Target Permissions
    • Management Console
      • KubeConfig Authenticaton
      • Authentication Methods Via Dex
      • UI Authentication
      • RBAC Authentication
      • Configuring the UI
    • Resource Request Requirements
      • Fine Tuning Resource Requests and Limits
    • Observability
      • Observability of Trilio with Prometheus and Grafana
      • Exported Prometheus Metrics
      • T4K Integration with Observability Stack
    • Modifying Default T4K Configuration
  • T4K Concepts
    • Supported Application Types
    • Support for Helm Releases
    • Support for OpenShift Operators
    • T4K Components
    • Backup and Restore Details
      • Immutable Backups
      • Application Centric Backups
    • Backup Retention Process
      • Retention Use Case
    • Continuous Restore
      • Architecture and Concepts
  • Performance
    • S3 as Backup Target
      • T4K S3 Fuse Plugin performance
    • Measuring Backup Performance
  • Ecosystem
    • T4K Integration with Slack using BotKube
    • Monitoring T4K Logs using ELK Stack
    • Rancher Navigation Links for Trilio Management Console
    • Optimize T4K Backups with StormForge
    • T4K GitHub Runner
    • AWS RDS snapshots using T4K hooks
    • Deploying Trilio For Kubernetes with Openshift ACM Policies
  • Krew Plugins
    • T4K QuickStart Plugin
    • Trilio for Kubernetes Preflight Checks Plugin
    • T4K Log Collector Plugin
    • T4K Cleanup Plugin
    • OCP ETCD Plugin
    • RKE ETCD Plugin
  • Support
    • Troubleshooting Guide
    • Known Issues and Workarounds
    • Contacting Support
  • Appendix
    • Ignored Resources
    • OpenSource Software Disclosure
    • CSI Drivers
      • Installing VolumeSnapshot CRDs
      • Install AWS EBS CSI Driver
    • T4K Product Quickview
    • OpenShift OperatorHub Custom CatalogSource
      • Custom CatalogSource in a restricted environment
    • Configure OVH Object Storage as a Target
    • Connect T4K UI hosted with HTTPS to another cluster hosted with HTTP or vice versa
    • Fetch DigitalOcean Kubernetes Cluster kubeconfig for T4K UI Authentication
    • Force Update T4K Operator in Rancher Marketplace
    • Backup and Restore Virtual Machines running on OpenShift
    • T4K For Volumes with Generic Storage
Powered by GitBook
On this page
  • About the Trilio Metrics Exporter for Prometheus
  • Consuming Metrics - Inside the Cluster
  • Using Annotations
  • Using Scrape Job
  • Using ServiceMonitor
  • Consuming Metrics - Outside the cluster
  • Visualizing Metrics with Grafana
  1. Advanced Configuration
  2. Observability

Observability of Trilio with Prometheus and Grafana

Learn how to consume and visualize performance metrics from the Trilio Metrics Exporter for Prometheus.

PreviousObservabilityNextExported Prometheus Metrics

About the Trilio Metrics Exporter for Prometheus

The Trilio Metrics Exporter is installed as a component of Trilio for Kubernetes. It is used to export Trilio performance metrics in the Prometheus format, consumable by your existing Prometheus monitoring. The exporter is based on the .

For details on the Prometheus formatted metrics provided by the Trilio Metrics Exporter, check the page.

In this How-to, we'll show you how you can consume performance metrics from the Trilio Metrics Exporter with your existing Prometheus installation.

In order to leverage the OpenShift built-in Prometheus instance hosted in the openshift-monitoring namespace, extra permissions will be required for the service account (openshift-monitoring:prometheus-k8s) to pull metrics from the openshift-operators namespace, where Trilio is deployed.

Consuming Metrics - Inside the Cluster

There are two methods to scrape the metrics from Trilio Exporter:

Using Annotations

Prometheus can discover any metric in the cluster exporter using the standard annotation set on the exporter pod.

The annotations are described below:

  • prometheus.io/scrape: The default configuration will scrape all pods; if set to false, this annotation will exclude the pod from the scraping process.

  • prometheus.io/path: If the metrics path is not /metrics, define it with this annotation.

  • prometheus.io/port: Scrape the pod on the indicated port instead of the pod’s declared ports; Here, it is 8080.

Trilio for Kubernetes has pre-configured the above annotations on the Metrics Exporter pod. If you have a cluster with Prometheus already installed and configured, then your Prometheus instance should start collecting Trilio for Kubernetes metrics without any configuration change.

Using Scrape Job

You will need to configure a Prometheus server to scrape the metrics from your newly running exporter. Add the following scrape job to your prometheus.yml configuration file.

  - job_name: trilio_exporter
    scrape_interval: 30s
    file_sd_configs:
    static_configs:
      - targets: ['EXPORTER_ADDRESS:8080']

EXPORTER_ADDRESS is the IP of the Prometheus exporter pod, which can be found by running the following

#kubectl get pod -o wide |grep -i export
k8s-triliovault-exporter-787c7dd446-bcgkr            1/1   Running   0     6d4h  192.168.140.126  slave2  <none>      <none>

Using ServiceMonitor

Step 1: Expose Trilio Exporter: To expose the Trilio metrics, you need to create a Kubernetes Service. If your Prometheus setup is not present in this cluster, expose it via a LoadBalancer Service.

You can create a LoadBalancer Service using the following.YAML

kind: Service
apiVersion: v1
metadata:
  name: k8s-triliovault-exporter
  labels:
    app: k8s-triliovault-exporter
spec:
  ports:
    - name: web
      protocol: TCP
      port: 8080
      targetPort: 8080
  selector:
    app: k8s-triliovault-exporter
  type: NodePort

Step 2. Create ServiceMonitor: You will need to configure a ServiceMonitor for Prometheus to scrape the metrics from your k8s-triliovault-exporter-service. Apply the following YAML to the namespace where T4K is installed.

  apiVersion: monitoring.coreos.com/v1
  kind: ServiceMonitor
  metadata:
    name: k8s-triliovault-exporter
    labels:
      app: k8s-triliovault-exporter
  spec:
    selector:
      matchLabels:
        app: k8s-triliovault-exporter
    endpoints:
    - port: web

Once the configuration is completed. You can find k8s-triliovault-exporter active targets in Prometheus UI.

Please wait a couple of minutes for Prometheus to start scraping metrics. Access Prometheus console at http://<prometheus-server-ip>/graph

If you leverage the Prometheus Operator stack, configure the ServiceMonitor the same way as the above configurations.

Consuming Metrics - Outside the cluster

To expose T4K metrics outside the cluster, perform the following two steps.

  1. Expose Trilio Exporter: Trilio metrics can be exposed outside the cluster via a LoadBalancer Service. Leverage kubectl to create a service for k8s-triliovault-exporter

kubectl expose deployment k8s-triliovault-exporter --type=LoadBalancer --name=exporter --namespace triliovault-integration

OR you can create a LoadBalancer Service using a .yaml file

kind: Service
apiVersion: v1
metadata:
  name: k8s-triliovault-exporter
  labels:
    app: k8s-triliovault-exporter
spec:
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
  selector:
    app: k8s-triliovault-exporter
  type: LoadBalancer

Once the Service is available, capture the Public IP for k8s-triliovault-exporter-service, this will be used in the second step.

  1. Scrape Job: You will need to configure a Prometheus server to scrape the metrics from your newly running exporter. Add the following scrape job to your prometheus.yml configuration file.

      - job_name: trilio_exporter
        scrape_interval: 30s
        file_sd_configs:
        static_configs:
          - targets: ['EXPORTER_PUBLIC_IP:8080']

Visualizing Metrics with Grafana

Metrics from Prometheus can be visualized leveraging Grafana. Dashboards can be created in Grafana with the Trilio metrics exposed through Prometheus.

Trilio Grafana Dashboards

Trilio provides pre-created Grafana dashboards to make monitoring and observing your backup landscape accessible.

Grafana dashboards are pivoted on the following themes and provide a high-level overview, summary, and details around each theme.

  1. Backups

  2. Restores

  3. Targets

  4. BackupPlans/Application

Import Grafana Dashboards

Please ensure that the dependencies mentioned for each dashboard are fulfilled. The T4K dashboards will not work as expected if these dependencies are unmet.

  1. Instructions for downloading and importing are provided with the dashboards.

T4K dashboards can be found on the Grafana within the

The dashboards can be imported into a Grafana instance following instructions from the Grafana .

official Prometheus client library
Exported Prometheus Metrics
Trilio org page.
project page
Trilio Prometheus Metrics
Grafana Dashboard Example