Observability of Trilio with Openshift Monitoring

Introduction

OpenShift Container Platform includes a built-in monitoring stack based on Prometheus that can be extended to monitor user workloads. This guide explains how to configure Trilio for Kubernetes (T4K) observability using OpenShift's native monitoring capabilities.

For more information about OpenShift monitoring, see the OpenShift Container Platform Monitoring Documentation.

Prerequisites

OpenShift Container Platform 4.x cluster
Trilio for Kubernetes installed
Cluster administrator access (for initial setup)
User with monitoring-edit or monitoring-rules-edit role (for configuring alerts)

Enabling User Workload Monitoring

Before you can monitor T4K metrics, you must enable user workload monitoring on your OpenShift cluster.

Enable User Workload Monitoring — Create ConfigMap

Create or edit the cluster-monitoring-config ConfigMap in the openshift-monitoring namespace:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    enableUserWorkload: true

Apply the configuration:

oc apply -f cluster-monitoring-config.yaml

Verify User Workload Monitoring is Running

Check that the user workload monitoring components are running:

oc get pods -n openshift-user-workload-monitoring

# Expected output:
# NAME                                   READY   STATUS    RESTARTS   AGE
# prometheus-operator-xxxxxxxxxx-xxxxx   2/2     Running   0          5m
# prometheus-user-workload-0             6/6     Running   0          5m
# prometheus-user-workload-1             6/6     Running   0          5m
# thanos-ruler-user-workload-0           4/4     Running   0          5m
# thanos-ruler-user-workload-1           4/4     Running   0          5m

Enable Alertmanager and AlertmanagerConfig

To use alerting with T4K, enable Alertmanager and allow users to create AlertmanagerConfig resources. There are two ConfigMaps to configure.

Configure Cluster Monitoring (openshift-monitoring)

Update the cluster-monitoring-config ConfigMap to enable user workload monitoring and allow AlertmanagerConfig resources to route to the platform Alertmanager:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    enableUserWorkload: true
    alertmanagerMain:
      enableUserAlertmanagerConfig: true

Apply the configuration:

oc apply -f cluster-monitoring-config.yaml

Configure User Workload Monitoring (openshift-user-workload-monitoring)

To enable a dedicated Alertmanager for user workload monitoring (separate from the platform Alertmanager), create the user-workload-monitoring-config ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    alertmanager:
      enabled: true
      enableAlertmanagerConfig: true

Apply the configuration:

oc apply -f user-workload-monitoring-config.yaml

Choosing between Platform and User Workload Alertmanager:

Configuration

Alertmanager Location

Use Case

cluster-monitoring-config with alertmanagerMain.enableUserAlertmanagerConfig: true

Platform (openshift-monitoring)

Route user alerts to the shared platform Alertmanager

user-workload-monitoring-config with alertmanager.enabled: true

User Workload (openshift-user-workload-monitoring)

Dedicated Alertmanager for user workloads, separate from platform alerts

You can use either or both configurations depending on your requirements.

Note: If you enable the user workload Alertmanager (user-workload-monitoring-config), AlertmanagerConfig resources in user namespaces will route to the user workload Alertmanager, not the platform Alertmanager.

Verify Alertmanager is Running

Platform Alertmanager (openshift-monitoring)

Check that the platform Alertmanager pods are running:

oc get pods -n openshift-monitoring -l app.kubernetes.io/name=alertmanager

# Expected output:
# NAME                   READY   STATUS    RESTARTS   AGE
# alertmanager-main-0    6/6     Running   0          5m
# alertmanager-main-1    6/6     Running   0          5m

Access the platform Alertmanager UI:

# Port-forward to access the UI
oc port-forward -n openshift-monitoring svc/alertmanager-main 9093:9093

# Open http://localhost:9093 in your browser

User Workload Alertmanager (openshift-user-workload-monitoring)

If you enabled the user workload Alertmanager, verify it's running:

oc get pods -n openshift-user-workload-monitoring -l app.kubernetes.io/name=alertmanager

# Expected output:
# NAME                            READY   STATUS    RESTARTS   AGE
# alertmanager-user-workload-0    6/6     Running   0          5m
# alertmanager-user-workload-1    6/6     Running   0          5m

Access the user workload Alertmanager UI:

# Port-forward to access the UI
oc port-forward -n openshift-user-workload-monitoring svc/alertmanager-user-workload 9093:9093

# Open http://localhost:9093 in your browser

Tip: You can check which Alertmanager your alerts are routing to by viewing the alert in the OpenShift Console under Observe > Alerting > Alerts and checking the source.

Grant User Permissions for Alert Routing (Optional)

To allow non-admin users to create AlertmanagerConfig resources, grant them the alert-routing-edit role:

# Grant alert-routing-edit to a specific user
oc adm policy add-role-to-user alert-routing-edit <username> -n <namespace>

# Or grant to a group
oc adm policy add-role-to-group alert-routing-edit <groupname> -n <namespace>

Available roles for monitoring:

Role

Description

monitoring-rules-view

View PrometheusRule and AlertmanagerConfig resources

monitoring-rules-edit

Create/modify PrometheusRule resources

monitoring-edit

Create/modify ServiceMonitor, PodMonitor, and PrometheusRule resources

alert-routing-edit

Create/modify AlertmanagerConfig resources

Configuring T4K Metrics Collection

Option 1: Enable ServiceMonitor via TVM Custom Resource (Recommended)

On OpenShift, enable Prometheus scraping for T4K by setting exporter.serviceMonitor.enabled: true in the TrilioVaultManager (TVM) Custom Resource.

Example TVM spec (edit your existing TVM and apply):

# you can update TVM CR from openshift OperatorHub console
apiVersion: triliovault.trilio.io/v1
kind: TrilioVaultManager
metadata:
  name: tvm
  namespace: trilio-system
spec:
  tvkInstanceName: tvk
  applicationScope: Cluster
  componentConfiguration:
    exporter:
      serviceMonitor:
        enabled: true

This creates:

A Service exposing the exporter metrics on port 8080 (created by the exporter)
A ServiceMonitor that configures Prometheus to scrape the metrics

Option 2: Create ServiceMonitor Manually

If you prefer to create the ServiceMonitor manually or need custom configuration, apply the following:

apiVersion: v1
kind: Service
metadata:
  name: k8s-triliovault-exporter-service
  namespace: <t4k-install-namespace>
  labels:
    app: k8s-triliovault-exporter
spec:
  ports:
    - name: web
      protocol: TCP
      port: 8080
      targetPort: 8080
  selector:
    app: k8s-triliovault-exporter
  type: ClusterIP
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: k8s-triliovault-exporter
  namespace: <t4k-install-namespace>
  labels:
    app: k8s-triliovault-exporter
spec:
  selector:
    matchLabels:
      app: k8s-triliovault-exporter
  endpoints:
    - port: web
      interval: 30s
      path: /metrics
      scheme: http

Apply the ServiceMonitor:

oc apply -f t4k-servicemonitor.yaml

When exporter.serviceMonitor.enabled is set to false (default), the exporter pod includes Prometheus scrape annotations (prometheus.io/scrape: "true"). If your Prometheus is configured to discover targets via annotations, metrics will be collected automatically without a ServiceMonitor.

Verifying Metrics Collection

After applying the ServiceMonitor, verify that metrics are being collected:

# Port-forward to the Thanos Querier
oc port-forward -n openshift-monitoring svc/thanos-querier 9090:9090

# In another terminal, query for T4K metrics
curl -s 'http://localhost:9090/api/v1/query?query=trilio_backup_info' | jq .

You can also verify from the OpenShift web console by navigating to Observe > Metrics and querying for trilio_backup_info.

Configuring T4K Alerting Rules

OpenShift uses PrometheusRule resources to define alerting rules. Create alerting rules for T4K in the namespace where T4K is installed, trilio-system by default.

T4K Alerting Rules

Create a file named t4k-prometheus-rules.yaml:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: t4k-alerting-rules
  namespace: trilio-system
  labels:
    app: k8s-triliovault
spec:
  groups:
    - name: t4k-backup-alerts
      rules:
        # Alert when backup fails (metric value -1 indicates Failed/Error status)
        - alert: T4KBackupFailed
          expr: trilio_backup_info == -1
          for: 1m
          labels:
            severity: critical
          annotations:
            summary: "TrilioVault Backup Failed"
            description: "Backup {{ $labels.backup }} has failed in namespace {{ $labels.resource_namespace }} on cluster {{ $labels.cluster }}"
        
        # Alert when backup is stuck in progress for too long
        - alert: T4KBackupStuck
          expr: trilio_backup_info{status="InProgress"} == 0 and trilio_backup_status_percentage < 100
          for: 60m
          labels:
            severity: warning
          annotations:
            summary: "TrilioVault Backup Stuck"
            description: "Backup {{ $labels.backup }} has been in progress for more than 60 minutes in namespace {{ $labels.resource_namespace }}. Current progress: {{ $value }}%"
        
        # Alert when backup takes unusually long time
        - alert: T4KBackupDurationHigh
          expr: trilio_backup_completed_duration > 120
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "TrilioVault Backup Duration High"
            description: "Backup {{ $labels.backup }} took {{ $value }} minutes to complete in namespace {{ $labels.resource_namespace }}"

    - name: t4k-restore-alerts
      rules:
        # Alert when restore fails
        - alert: T4KRestoreFailed
          expr: trilio_restore_info == -1
          for: 1m
          labels:
            severity: critical
          annotations:
            summary: "TrilioVault Restore Failed"
            description: "Restore {{ $labels.restore }} has failed in namespace {{ $labels.resource_namespace }} on cluster {{ $labels.cluster }}"
        
        # Alert when restore is stuck
        - alert: T4KRestoreStuck
          expr: trilio_restore_info{status="InProgress"} == 0 and trilio_restore_status_percentage < 100
          for: 60m
          labels:
            severity: warning
          annotations:
            summary: "TrilioVault Restore Stuck"
            description: "Restore {{ $labels.restore }} has been in progress for more than 60 minutes in namespace {{ $labels.resource_namespace }}"

    - name: t4k-target-alerts
      rules:
        # Alert when target is unavailable (metric value 0 indicates unavailable)
        - alert: T4KTargetUnavailable
          expr: trilio_target_info == 0
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "TrilioVault Target Unavailable"
            description: "Target {{ $labels.target }} is not available in namespace {{ $labels.resource_namespace }}. Status: {{ $labels.status }}"
        
        # Alert when target storage exceeds threshold (example: 500GB)
        - alert: T4KTargetStorageHigh
          expr: trilio_target_storage > 500000000000
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "TrilioVault Target Storage High"
            description: "Target {{ $labels.target }} storage usage is {{ $value | humanize1024 }}B in namespace {{ $labels.resource_namespace }}"

    - name: t4k-backupplan-alerts
      rules:
        # Alert when BackupPlan has no successful backups (not protected)
        - alert: T4KBackupPlanNotProtected
          expr: trilio_backupplan_info{protected="False"} == 1
          for: 24h
          labels:
            severity: warning
          annotations:
            summary: "TrilioVault BackupPlan Not Protected"
            description: "BackupPlan {{ $labels.backupplan }} in namespace {{ $labels.resource_namespace }} has no successful backups for more than 24 hours"
        
        # Alert when BackupPlan fails
        - alert: T4KBackupPlanFailed
          expr: trilio_backupplan_info == -1
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "TrilioVault BackupPlan Failed"
            description: "BackupPlan {{ $labels.backupplan }} has failed in namespace {{ $labels.resource_namespace }}"

    - name: t4k-continuous-restore-alerts
      rules:
        # Alert when ContinuousRestorePlan fails
        - alert: T4KContinuousRestorePlanFailed
          expr: trilio_continuousrestoreplan_info == -1
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "TrilioVault ContinuousRestorePlan Failed"
            description: "ContinuousRestorePlan {{ $labels.continuousrestoreplan }} has failed on cluster {{ $labels.cluster }}"
        
        # Alert when ConsistentSet fails
        - alert: T4KConsistentSetFailed
          expr: trilio_consistentset_info == -1
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "TrilioVault ConsistentSet Failed"
            description: "ConsistentSet {{ $labels.consistentset }} has failed for ContinuousRestorePlan {{ $labels.continuousrestoreplan }}"

Apply the alerting rules:

oc apply -f t4k-prometheus-rules.yaml

Verifying Alerting Rules

Check that the alerting rules are loaded:

oc get prometheusrules -n <t4k-install-namespace>

# View the rule details
oc describe prometheusrule t4k-alerting-rules -n <t4k-install-namespace>

You can also view the alerts in the OpenShift web console by navigating to Observe > Alerting > Alerting rules.

Configuring Alert Routing

OpenShift supports AlertmanagerConfig resources for configuring alert routing in user workloads. This allows you to define custom receivers and routing rules for T4K alerts.

Prerequisites for Alert Routing

Ensure your user has the alert-routing-edit cluster role:

oc adm policy add-cluster-role-to-user alert-routing-edit <username> -n <t4k-install-namespace>

Example: AlertmanagerConfig with Slack Notifications

Create a file named t4k-alertmanager-config.yaml:

apiVersion: monitoring.coreos.com/v1beta1
kind: AlertmanagerConfig
metadata:
  name: t4k-alert-routing
  namespace: <t4k-install-namespace>
  labels:
    app: k8s-triliovault
spec:
  route:
    receiver: default
    groupBy:
      - alertname
      - namespace
    groupWait: 30s
    groupInterval: 5m
    repeatInterval: 4h
    routes:
      - receiver: slack-critical
        matchers:
          - name: severity
            value: critical
            matchType: "="
      - receiver: slack-warning
        matchers:
          - name: severity
            value: warning
            matchType: "="
  
  receivers:
    - name: default
    
    - name: slack-critical
      slackConfigs:
        - apiURL:
            name: slack-webhook-secret
            key: webhook-url
          channel: '#t4k-critical-alerts'
          sendResolved: true
          title: '[{{ .Status | toUpper }}] {{ .GroupLabels.alertname }}'
          text: |-
            {{ range .Alerts }}
            *Alert:* {{ .Annotations.summary }}
            *Description:* {{ .Annotations.description }}
            *Severity:* {{ .Labels.severity }}
            *Namespace:* {{ .Labels.resource_namespace }}
            {{ end }}
    
    - name: slack-warning
      slackConfigs:
        - apiURL:
            name: slack-webhook-secret
            key: webhook-url
          channel: '#t4k-warning-alerts'
          sendResolved: true

Creating the Slack Webhook Secret

Store your Slack webhook URL in a Kubernetes Secret:

oc create secret generic slack-webhook-secret \
  --from-literal=webhook-url='https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK' \
  -n <t4k-install-namespace>

Apply the AlertmanagerConfig:

oc apply -f t4k-alertmanager-config.yaml

Example: AlertmanagerConfig with Email Notifications

apiVersion: monitoring.coreos.com/v1beta1
kind: AlertmanagerConfig
metadata:
  name: t4k-email-alerts
  namespace: <t4k-install-namespace>
  labels:
    app: k8s-triliovault
spec:
  route:
    receiver: email-receiver
    groupBy:
      - alertname
    groupWait: 30s
    groupInterval: 5m
    repeatInterval: 4h
  
  receivers:
    - name: email-receiver
      emailConfigs:
        - to: '[email protected]'
          from: '[email protected]'
          smarthost: 'smtp.example.com:587'
          authUsername: '[email protected]'
          authPassword:
            name: smtp-secret
            key: password
          sendResolved: true

Create the SMTP password secret:

oc create secret generic smtp-secret \
  --from-literal=password='your-smtp-password' \
  -n <t4k-install-namespace>

Example: AlertmanagerConfig with PagerDuty

apiVersion: monitoring.coreos.com/v1beta1
kind: AlertmanagerConfig
metadata:
  name: t4k-pagerduty-alerts
  namespace: <t4k-install-namespace>
  labels:
    app: k8s-triliovault
spec:
  route:
    receiver: pagerduty-critical
    groupBy:
      - alertname
      - namespace
    routes:
      - receiver: pagerduty-critical
        matchers:
          - name: severity
            value: critical
            matchType: "="
  
  receivers:
    - name: pagerduty-critical
      pagerdutyConfigs:
        - serviceKey:
            name: pagerduty-secret
            key: service-key
          severity: critical
          description: '{{ .CommonAnnotations.summary }}'
          details:
            - key: namespace
              value: '{{ .CommonLabels.resource_namespace }}'
            - key: alertname
              value: '{{ .CommonLabels.alertname }}'

Create the PagerDuty secret:

oc create secret generic pagerduty-secret \
  --from-literal=service-key='YOUR_PAGERDUTY_SERVICE_KEY' \
  -n <t4k-install-namespace>

Custom Notification Templates

OpenShift Alertmanager supports custom notification templates for email and Slack messages. This section explains how to configure T4K-specific templates with rich formatting.

Use inline templates directly in your AlertmanagerConfig for full control over notification formatting:

apiVersion: monitoring.coreos.com/v1beta1
kind: AlertmanagerConfig
metadata:
  name: t4k-custom-alerts
  namespace: <t4k-install-namespace>
  labels:
    app: k8s-triliovault
spec:
  route:
    receiver: t4k-alerts
    groupBy:
      - alertname
      - backup
      - namespace
    groupWait: 30s
    groupInterval: 5m
    repeatInterval: 4h
    matchers:
      - name: alertname
        matchType: =~
        value: "TVK.*"
  
  receivers:
    - name: t4k-alerts
      emailConfigs:
        - to: '[email protected]'
          from: '[email protected]'
          smarthost: 'smtp.example.com:587'
          authUsername: 'user'
          authPassword:
            name: t4k-smtp-secret
            key: password
          sendResolved: false
          # Custom email subject
          headers:
            - key: Subject
              value: '[{{ .Status | toUpper }}] TrilioVault: {{ .CommonLabels.alertname }}'
          # Custom HTML email body
          html: |
            <!DOCTYPE html>
            <html>
            <head><style>
              body { font-family: Arial, sans-serif; padding: 20px; max-width: 800px; margin: 0 auto; }
              .header { padding: 15px; background: {{ if eq .CommonLabels.alertname "T4KBackupFailed" }}#f44336{{ else }}#4CAF50{{ end }}; color: white; border-radius: 4px; }
              .content { padding: 20px; background: #f5f5f5; margin-top: 10px; border-radius: 4px; }
              table { width: 100%; border-collapse: collapse; margin: 15px 0; }
              th { background: #34495e; color: white; padding: 10px; text-align: left; }
              td { padding: 10px; border-bottom: 1px solid #ddd; background: white; }
              .footer { margin-top: 20px; color: #666; font-size: 12px; }
            </style></head>
            <body>
              <div class="header">
                <h1>TrilioVault: {{ .CommonLabels.alertname }}</h1>
              </div>
              {{ range .Alerts }}
              <div class="content">
                <h3>{{ .Annotations.summary }}</h3>
                <p>{{ .Annotations.description }}</p>
                <table>
                  <tr><th>Property</th><th>Value</th></tr>
                  <tr><td>Backup</td><td>{{ .Labels.backup }}</td></tr>
                  <tr><td>Kind</td><td>{{ .Labels.kind }}</td></tr>
                  <tr><td>BackupPlan</td><td>{{ .Labels.backupplan }}</td></tr>
                  <tr><td>Namespace</td><td>{{ .Labels.namespace }}</td></tr>
                  <tr><td>Target</td><td>{{ .Labels.target }}</td></tr>
                  <tr><td>Status</td><td>{{ .Labels.status }}</td></tr>
                  <tr><td>Cluster</td><td>{{ .Labels.cluster }}</td></tr>
                </table>
              </div>
              {{ end }}
              <div class="footer">
                <p>Severity: {{ .CommonLabels.severity }}</p>
              </div>
            </body>
            </html>
      
      slackConfigs:
        - apiURL:
            name: t4k-slack-secret
            key: webhook-url
          channel: '#t4k-alerts'
          sendResolved: false
          # Custom Slack title with emoji indicators
          title: '{{ if eq .CommonLabels.alertname "T4KBackupFailed" }}:x:{{ else }}:white_check_mark:{{ end }} [{{ .Status | toUpper }}] {{ .CommonLabels.alertname }}'
          # Custom Slack message body
          text: |
            {{ if eq .CommonLabels.alertname "T4KBackupFailed" }}:rotating_light: *Backup Failed*{{ else }}:tada: *Backup Successful*{{ end }}
            {{ range .Alerts }}
            > *{{ .Annotations.summary }}*
            > {{ .Annotations.description }}

            *Backup Details*
            • Backup: `{{ .Labels.backup }}`
            • Kind: `{{ .Labels.kind }}`
            • BackupPlan: `{{ .Labels.backupplan }}`
            • Namespace: `{{ .Labels.namespace }}`
            • Target: `{{ .Labels.target }}`
            • Status: `{{ .Labels.status }}`
            • Cluster: `{{ .Labels.cluster }}`
            • Severity: `{{ .Labels.severity }}`
            {{ end }}
          # Dynamic color based on alert type
          color: '{{ if eq .CommonLabels.alertname "T4KBackupFailed" }}danger{{ else }}good{{ end }}'
---
apiVersion: v1
kind: Secret
metadata:
  name: t4k-smtp-secret
  namespace: <t4k-install-namespace>
type: Opaque
stringData:
  password: "your-smtp-password"
---
apiVersion: v1
kind: Secret
metadata:
  name: t4k-slack-secret
  namespace: <t4k-install-namespace>
type: Opaque
stringData:
  webhook-url: "https://hooks.slack.com/services/YOUR/WEBHOOK/URL"

Viewing Metrics and Alerts in OpenShift Console

Viewing Metrics

Navigate to Observe > Metrics in the OpenShift web console
Select your project namespace from the dropdown
Enter a PromQL query, for example:
- trilio_backup_info - View all backup statuses
- trilio_backup_info == -1 - View failed backups
- trilio_target_info - View target statuses

Viewing Alerts

Navigate to Observe > Alerting in the OpenShift web console
Click on Alerting rules to see all configured rules including T4K rules
Click on Alerts to see currently firing alerts
Use filters to narrow down to T4K alerts by searching for "T4K"

Viewing Alert Silences

Navigate to Observe > Alerting > Silences
Create silences for maintenance windows or known issues

T4K Metrics Reference

Trilio for Kubernetes exports the following Prometheus metrics:

Metric Value Conventions

For status-based metrics (*_info metrics), the numeric value indicates the status:

Status

Metric Value

Description

Available / Completed

1

Resource is healthy/successful

Failed / Error

-1

Resource has failed

InProgress

0

Operation is in progress

Empty/Unknown

-2

Status not yet determined

Available Metrics

Backup Metrics

Metric Name

Description

trilio_backup_info

Backup status and metadata

trilio_backup_storage

Backup size in bytes

trilio_backup_status_percentage

Backup progress (0-100)

trilio_backup_completed_duration

Backup duration in minutes (only for completed backups)

trilio_backup_metadata_info

Detailed backup object metadata

Restore Metrics

Metric Name

Description

trilio_restore_info

Restore status and metadata

trilio_restore_status_percentage

Restore progress (0-100)

trilio_restore_completed_duration

Restore duration in minutes (only for completed restores)

Target Metrics

Metric Name

Description

trilio_target_info

Target availability status (1=available, 0=unavailable)

trilio_target_storage

Storage used by target in bytes

BackupPlan Metrics

Metric Name

Description

trilio_backupplan_info

BackupPlan status and summary

trilio_backupplan_crstatus

BackupPlan continuous restore status

Continuous Restore Metrics

Metric Name

Description

trilio_continuousrestoreplan_info

ContinuousRestorePlan status

trilio_consistentset_info

ConsistentSet status and details

trilio_consistentset_status_percentage

ConsistentSet progress (0-100)

trilio_consistentset_completed_duration

ConsistentSet duration in minutes

Example PromQL Queries

# Count of failed backups by namespace
count(trilio_backup_info == -1) by (resource_namespace)

# List all successful backups
trilio_backup_info{status="Available"}

# Total backup storage per target
sum(trilio_backup_storage) by (target)

# Average backup duration by backupplan
avg(trilio_backup_completed_duration) by (backupplan)

# Unavailable targets
trilio_target_info == 0

# BackupPlans without successful backups
trilio_backupplan_info{protected="False"}

# Failed restores
trilio_restore_info == -1

# ContinuousRestorePlan replication status
trilio_consistentset_info{status="InProgress"}

Separating Platform and User Alerts

OpenShift adds the label openshift_io_alert_source="platform" to all platform alerts. You can use this to configure different routing for T4K alerts:

apiVersion: monitoring.coreos.com/v1beta1
kind: AlertmanagerConfig
metadata:
  name: t4k-user-alerts-routing
  namespace: <t4k-install-namespace>
spec:
  route:
    receiver: t4k-alerts-receiver
    matchers:
      # Match only user-defined alerts (not platform alerts)
      - name: openshift_io_alert_source
        value: platform
        matchType: "!="
    groupBy:
      - alertname
      - namespace
  
  receivers:
    - name: t4k-alerts-receiver
      slackConfigs:
        - apiURL:
            name: slack-webhook-secret
            key: webhook-url
          channel: '#t4k-alerts'
          sendResolved: true

Troubleshooting

Metrics Not Appearing

Verify the T4K exporter pod is running:

oc get pods -n <t4k-install-namespace> -l app.kubernetes.io/name=k8s-triliovault-exporter

Check if the ServiceMonitor is correctly configured:
```
oc get servicemonitor -n <t4k-install-namespace>
```

Verify Prometheus is scraping the target:

oc port-forward -n openshift-user-workload-monitoring svc/prometheus-user-workload 9090:9090
# Then visit http://localhost:9090/targets

Alerts Not Firing

Verify the PrometheusRule is loaded:

oc get prometheusrules -n <t4k-install-namespace>

Check for errors in the Prometheus logs:

oc logs -n openshift-user-workload-monitoring -l app.kubernetes.io/name=prometheus -c prometheus

Verify the alert expression returns results:

# Use the Metrics console to test your expression
# Navigate to Observe > Metrics and run: trilio_backup_info == -1

Alert Notifications Not Received

Verify the AlertmanagerConfig is applied:

oc get alertmanagerconfig -n <t4k-install-namespace>

Check the Alertmanager logs:

oc logs -n openshift-user-workload-monitoring -l app.kubernetes.io/name=alertmanager

Verify secrets are correctly created:

oc get secrets -n <t4k-install-namespace> | grep -E 'slack|smtp|pagerduty'

Additional Resources

PreviousT4K Integration with Observability Stack NextModifying Default T4K Configuration

Last updated 1 day ago

hashtagIntroduction

hashtagPrerequisites

hashtagEnabling User Workload Monitoring

hashtagEnable User Workload Monitoring — Create ConfigMap

hashtagVerify User Workload Monitoring is Running

hashtagEnable Alertmanager and AlertmanagerConfig

hashtagConfigure Cluster Monitoring (openshift-monitoring)

hashtagConfigure User Workload Monitoring (openshift-user-workload-monitoring)

hashtagVerify Alertmanager is Running

hashtagPlatform Alertmanager (openshift-monitoring)

hashtagUser Workload Alertmanager (openshift-user-workload-monitoring)

hashtagGrant User Permissions for Alert Routing (Optional)

hashtagConfiguring T4K Metrics Collection

hashtagOption 1: Enable ServiceMonitor via TVM Custom Resource (Recommended)

hashtagOption 2: Create ServiceMonitor Manually

hashtagVerifying Metrics Collection

hashtagConfiguring T4K Alerting Rules

hashtagT4K Alerting Rules

hashtagVerifying Alerting Rules

hashtagConfiguring Alert Routing

hashtagPrerequisites for Alert Routing

hashtagExample: AlertmanagerConfig with Slack Notifications

hashtagCreating the Slack Webhook Secret

hashtagExample: AlertmanagerConfig with Email Notifications

hashtagExample: AlertmanagerConfig with PagerDuty

hashtagCustom Notification Templates

hashtagViewing Metrics and Alerts in OpenShift Console

hashtagViewing Metrics

hashtagViewing Alerts

hashtagViewing Alert Silences

hashtagT4K Metrics Reference

hashtagMetric Value Conventions

hashtagAvailable Metrics

hashtagBackup Metrics

hashtagRestore Metrics

hashtagTarget Metrics

hashtagBackupPlan Metrics

hashtagContinuous Restore Metrics

hashtagExample PromQL Queries

hashtagSeparating Platform and User Alerts

hashtagTroubleshooting

hashtagMetrics Not Appearing

hashtagAlerts Not Firing

hashtagAlert Notifications Not Received

hashtagAdditional Resources

Introduction

Prerequisites

Enabling User Workload Monitoring

Enable User Workload Monitoring — Create ConfigMap

Verify User Workload Monitoring is Running

Enable Alertmanager and AlertmanagerConfig

Configure Cluster Monitoring (openshift-monitoring)

Configure User Workload Monitoring (openshift-user-workload-monitoring)

Verify Alertmanager is Running

Platform Alertmanager (openshift-monitoring)

User Workload Alertmanager (openshift-user-workload-monitoring)

Grant User Permissions for Alert Routing (Optional)

Configuring T4K Metrics Collection

Option 1: Enable ServiceMonitor via TVM Custom Resource (Recommended)

Option 2: Create ServiceMonitor Manually

Verifying Metrics Collection

Configuring T4K Alerting Rules

T4K Alerting Rules

Verifying Alerting Rules

Configuring Alert Routing

Prerequisites for Alert Routing

Example: AlertmanagerConfig with Slack Notifications

Creating the Slack Webhook Secret

Example: AlertmanagerConfig with Email Notifications

Example: AlertmanagerConfig with PagerDuty

Custom Notification Templates

Viewing Metrics and Alerts in OpenShift Console

Viewing Metrics

Viewing Alerts

Viewing Alert Silences

T4K Metrics Reference

Metric Value Conventions

Available Metrics

Backup Metrics

Restore Metrics

Target Metrics

BackupPlan Metrics

Continuous Restore Metrics

Example PromQL Queries

Separating Platform and User Alerts

Troubleshooting

Metrics Not Appearing

Alerts Not Firing

Alert Notifications Not Received

Additional Resources