# Grafana Monitoring Installation and Configuration on RHOSO18 ### 1] Clone triliovault-cfg-scripts Repository The following steps must be performed on the `bastion` node of an already installed RHOSO environment. Run the following command to clone the triliovault-cfg-scripts GitHub repository: ``` git clone -b 6.1.7-1 https://github.com/trilioData/triliovault-cfg-scripts.git cd triliovault-cfg-scripts/redhat-director-scripts/monitoring-openshift/ ``` *** ## 2] Enable User Workload Monitoring in OpenShift ### 2.1] Check if user workload monitoring is already enabled on OpenShift * Check if the following ConfigMap exists: ```bash oc get cm -n openshift-monitoring | grep cluster-monitoring-config ``` * If it exists, check whether the `enableUserWorkload: true` parameter is present with its value set to `true` under `config.yaml`, as shown below. If it is present, user workload monitoring is enabled. If it is not present or is set to `false`, you must enable it using the next steps. Otherwise, skip the steps to enable user workload monitoring. ```bash oc get -n openshift-monitoring cm cluster-monitoring-config -o yaml ``` ```yaml apiVersion: v1 data: config.yaml: | enableUserWorkload: true kind: ConfigMap metadata: name: cluster-monitoring-config namespace: openshift-monitoring ``` *** ### 2.2] If user workload monitoring is not enabled, enable it * Check whether the following ConfigMap already exists. If it does not exist, create it using the following YAML file. No changes are required in the YAML file. ```bash cd triliovault-cfg-scripts/redhat-director-scripts/monitoring-openshift/ oc apply -f cm-cluster-monitoring-config.yaml -n openshift-monitoring ``` * If the ConfigMap `cluster-monitoring-config` already exists, add the following parameter to it: ```yaml config.yaml: | enableUserWorkload: true ``` * Edit the ConfigMap using: ```bash oc edit cm cluster-monitoring-config -n openshift-monitoring ``` *** ## 3] Steps to Install Prometheus SQL Exporter ### 3.1] Create SQL Exporter ConfigMap * First, you need to create a ConfigMap that holds the SQL Exporter's configuration. This includes the database connection string. * Edit `sql-exporter-config.yaml`. Before applying, locate the `data_source_name` field and update it with the correct MySQL username, password, and hostname for your database. * **Important:** Do not change the database name or the overall format of the `data_source_name` string. Only update credentials and hostname. ```bash cd triliovault-cfg-scripts/redhat-director-scripts/monitoring-openshift/sql_exporters/ vi sql-exporter-config.yaml ``` * Create the Secret. Before applying the deployment, create the secret containing the DSN string: ```bash oc -n openshift-user-workload-monitoring create secret generic sql-exporter-secret --from-literal=dsn="mysql://:@:3306/workloadmgr" ``` Replace ``, ``, and `` with the actual values. * Apply the ConfigMap: ```bash oc -n openshift-user-workload-monitoring apply -f sql-exporter-config.yaml ``` *** ### 3.2] Apply SQL Exporter Deployment * Deploy the SQL Exporter application using its Deployment manifest: ```bash oc -n openshift-user-workload-monitoring apply -f sql-exporter-deployment.yaml ``` * Verify pod status after deployment: ```bash oc -n openshift-user-workload-monitoring get pods | grep sql ``` You should see a pod with the status `Running`. *** ### 3.3] Apply SQL Exporter Service * Create a Kubernetes Service to expose the SQL Exporter within the cluster so that other services (such as Prometheus) can discover and scrape its metrics: ```bash oc -n openshift-user-workload-monitoring apply -f sql-exporter-service.yaml ``` * Verify Service Status: Confirm that the Service has been created. ```bash oc get svc -n openshift-user-workload-monitoring | grep sql ``` *** ### 3.4] Apply SQL Exporter ServiceMonitor Finally, create a ServiceMonitor resource. This tells Prometheus (if configured to discover ServiceMonitors in this namespace) to automatically scrape metrics from your SQL Exporter Service. ```bash oc -n openshift-user-workload-monitoring apply -f sql-exporter-servicemonitor.yaml ``` *** ## 4] Steps to Install Prometheus OpenStack Exporter Follow below steps to deploy the OpenStack Exporter: ### 4.1] Create ConfigMap with OpenStack Cloud Credentials First, create a ConfigMap containing your OpenStack cloud credentials. This ConfigMap will be mounted into the exporter pod. Replace all placeholder variables (such as auth URL, username, password, etc.) with actual values from your OpenStack cloud in the following config file. ```bash cd triliovault-cfg-scripts/redhat-director-scripts/monitoring-openshift/openstack_exporter/ # Fill in OpenStack cloud admin user credentials in the following YAML file vi openstack-exporter-clouds-cm.yaml # Save changes # Apply the changes oc -n openshift-user-workload-monitoring apply -f openstack-exporter-clouds-cm.yaml ``` *** ### 4.2] Create Keystone CA Certificate Secret Before deploying the OpenStack Exporter, create the required secret containing the Keystone CA certificate. ```bash cd triliovault-cfg-scripts/redhat-director-scripts/monitoring-openshift/openstack_exporter/ ./create_keystone_cacert_secret.sh ``` *** ### 4.3] Deploy OpenStack Exporter Next, deploy the OpenStack Exporter application using its Deployment manifest. ```bash oc -n openshift-user-workload-monitoring apply -f openstack-exporter-deployment.yaml ``` *** ### 4.4] Verify OpenStack Exporter Pod Status After deployment, verify that the exporter pod is running correctly. ```bash oc -n openshift-user-workload-monitoring get pods | grep openstack-exporter ``` You should see a pod with a status of `Running`. *** ### 4.5] Check OpenStack Exporter Pod Logs To ensure the exporter is starting and connecting to OpenStack successfully, check its logs. Replace `` with the actual name of your exporter pod obtained from the previous step. ```bash oc -n openshift-user-workload-monitoring logs ``` Look for messages indicating a successful connection to OpenStack and that it is listening for metrics requests (for example, on port 9180). *** ### 4.6] Create Service for OpenStack Exporter Create a Kubernetes Service to expose the OpenStack Exporter within the cluster, allowing other services (like Prometheus) to discover and scrape its metrics. ```bash oc -n openshift-user-workload-monitoring apply -f openstack-exporter-service.yaml ``` *** ### 4.7] Create ServiceMonitor for Prometheus Integration Finally, create a ServiceMonitor resource. This tells Prometheus (if configured to discover ServiceMonitors in this namespace) to automatically scrape metrics from your OpenStack Exporter Service. ```bash oc -n openshift-user-workload-monitoring apply -f openstack-exporter-servicemonitor.yaml ``` *** ## 5] Install and Configure Grafana ### 5.1] Install Grafana Operator 1. Go to **Administrator View → Operators → OperatorHub**. 2. Search for **Grafana Operator** and select it. 3. Click **Install** to open the operator installation page. 4. In the installation configuration: * Choose the appropriate **Update Channel**. * Select the **latest available operator version**. * Under **Installation Mode**, select: * **A specific namespace on the cluster** * Select **Installed Namespace** to: ``` openshift-user-workload-monitoring ``` * Keep **Update Approval** as **Automatic** (recommended). 5. Click **Install** to confirm the installation. > **Note:** It is recommended to install the Grafana Operator in the `openshift-user-workload-monitoring` namespace, as this namespace is used for user workload monitoring in RHOSO. *** ### 5.2] Create Grafana service account ```bash oc -n openshift-user-workload-monitoring create sa grafana-sa oc -n openshift-user-workload-monitoring adm policy add-cluster-role-to-user cluster-monitoring-view -z grafana-sa ``` *** ### 5.3] Create credential secret for Grafana Edit the following secret file and set fresh username and password values. Token must be fetched using the following command: ```bash BEARER_TOKEN=$(oc create token grafana-sa --duration=87600h -n openshift-user-workload-monitoring | base64 -w0) cp grafana-secret-creds_original.yaml grafana-secret-creds.yaml sed -i "s/\$BASE64_ENCODED_GRAFANA_SERVICE_ACCOUNT_BEARER_TOKEN/${BEARER_TOKEN}/g" grafana-secret-creds.yaml # Using above sed command, token got copied to grafan secret yaml file ``` After running the above command, the token is copied into the Grafana secret YAML file. ```bash # Set username and password and save changes vi grafana-secret-creds.yaml # Apply the secret oc -n openshift-user-workload-monitoring apply -f grafana-secret-creds.yaml ``` *** ### 5.4] Create Grafana instance For a fresh deployment, create a Grafana instance: ```bash oc -n openshift-user-workload-monitoring apply -f grafana-instance.yaml ``` In case of a token update on an existing Grafana instance, this is not required if you are installing Grafana for the first time: ```bash oc rollout restart deployment grafana-deployment -n openshift-user-workload-monitoring ``` *** ### 5.5] Verify Grafana installation ```bash $ oc -n openshift-user-workload-monitoring get GRAFANA NAME VERSION STAGE STAGE STATUS AGE grafana 11.3.0 complete success 49s $ oc -n openshift-user-workload-monitoring get pods -l app=grafana NAME READY STATUS RESTARTS AGE grafana-deployment-645b4bbb56-5pj6x 1/1 Running 0 4m43s $ oc -n openshift-user-workload-monitoring get route grafana -o jsonpath='{.spec.host}' grafana-openshift-user-workload-monitoring.apps.roso18qa.prod.engineering.trilio.io ``` *** ### 5.6] Expose Grafana service if route not found If the above command to get the Grafana route returns a **NotFound** error, you need to expose the Grafana service manually to create the route: ```bash oc -n openshift-user-workload-monitoring expose svc/grafana-service --name=grafana --port=grafana oc -n openshift-user-workload-monitoring get route grafana -o jsonpath='{.spec.host}' ``` *** ### 5.7] Create Grafana datasource Create Grafana datasource to fetch metrics from the cluster monitoring namespace ```bash oc -n openshift-user-workload-monitoring apply -f grafana-datasource.yaml ``` Verify ```bash oc -n openshift-user-workload-monitoring get GRAFANADATASOURCE ``` *** --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.trilio.io/openstack/admin-guide/grafana-monitoring/rhoso18.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.