Known Issues and Workarounds

This page describes issues that may be encountered and potential solutions to those issues.

Management Console not accessible using NodePort

If user has done all the configuration mentioned as per the Accessing the Management Console instructions and still not able to access the UI, check the Firewall Rules for the Kubernetes cluster nodes.

Here is an example to check firewall rules on the Google GKE cluster:

1. Search the VPC Network on GCP web console

2. Select `Firewall` option from the left pane

Select Firewall from VPC Network

1. Search the rules with your Kubernetes cluster name

2. In the list of firewalls, verify:

   1. Filters column - Shows the source IP which can access the T4K Management Console Hostname and NodePort

   2. Protocols/Ports column - Shows the ports which are accessible on the cluster nodes.

3. Verify that the NodePort assigned to service k8s-triliovault-ingress-gateway is added in the firewall rules Protocols and Ports.

Filters showing IP range and Protols/Ports showing exposed Port

Target Creation Failure

kubectl apply -f <backup_target.yaml> is not successful and the resource goes into Unavailable state.

Possible reasons for the target marked Unavailable could be not having enough permissions or lack of network connectivity. Target resource creation launches a validation pod to check if Trilio can connect and write to the Target. Trilio then cleans the pod after the validation process completes, which clears any logs that the validation pod has. Users should repeat the same operation and actively check the logs on the validation pods.

Kubernetes

kubectl get pods -A | grep validator
kubectl logs <validator-pod> -n <namespace>

OpenShift

oc get pods -A | grep validator
oc logs <validator-pod> -n <namespace

Backup Failures

Please follow the following steps to analyze backup failures.

kubectl get jobs
oc get jobs

To troubleshoot backup issues, it is important to know the different phases of jobs.triliovault pod.

1. <backupname>-snapshot: Trilio leverages CSI snapshot functionality to take PV snapshots. If the backup fails in this step user should try a manual snapshot of the PV to make sure all required drivers are present, and that this operation works without Trilio.

2. <backupname>-upload: In this phase, Trilio uploads metadata and data to the Target. If the backup fails in this phase, we need to look into the following logs for error.

Kubernetes

kubectl get pods -A | grep k8s-triliovault-control-plane
kubectl logs <controller POD>
kubectl logs <failed or errored datamover pod>

OpenShift

oc get pods -A | grep k8s-triliovault-control-plane
oc logs <controller POD>
oc logs <failed or errored datamover pod>

Restore Failures

The most common reasons for restore failures are resource conflicts. Users may try to restore the backup to the same namespace where the original application and restored application have names conflict. Please check the logs to analyze the root cause of the issue.

Kubernetes

kubectl get pods -A | grep k8s-triliovault-control-plane
kubectl logs <controller POD>
kubectl logs <failed or errored metamoved pod>

OpenShift

oc get pods -A | grep k8s-triliovault-control-plane
oc logs <controller POD>
oc logs <failed or errored metamoved pod>

Webhooks are installed

Problem Description: If a Trilio operation is failing with an error Internal error occurred: failed or service "k8s-triliovault-webhook-service" not found

Error from server (InternalError): error when creating "targetmigration.yaml": Internal error occurred: failed 
calling webhook "mapplication.kb.io": Post https://k8s-triliovault-webhook-service.default.svc:443/mutate?timeout=30s: service "k8s-triliovault-webhook-service" not found

Solution : This might be due to the installation of T4K in multiple Namespaces. Even a simple clean-up of those installations will not have webhook instances cleared. Please run the following commands to get all the webhooks instances.

kubectl get validatingwebhookconfigurations  -A | grep trilio
kubectl get mutatingwebhookconfigurations -A | grep trilio

Delete duplicate entries if any.

Restore Failed

Problem Description: If you are not able to restore an application into different namespace (OpenShift)

Error:
Permission denied: could not bind to address 0.0.0.0:80

Solution: User may not have sufficient permissions to restore the application into the namespace.

oc adm policy add-scc-to-user anyuid -z default -n <NameSpace>O

OOMKilled Issues

Sometimes users may notice some Trilio pods go into OOMKilled state. This can happen in environments where a lot of objects exist for which additional processing power may be needed. In such scenarios, the memory and CPU limits can be bumped up by running the following commands:

Kubernetes

Please follow the API documentation to increase resource limits for Trilio pods deployed via the upstream Helm based Operator

OCP

Within OCP, the memory and CPU values for Trilio are managed and controlled by the Operator ClusterServiceVersion(CSV)Users can either edit the Operator CSV YAML directly from within the OCP console or can patch the pods based on the snippets provided below.

The following snippet shows how to increase the memory value to 2048Mi for the web-backend pod

export VERSION=2.6.0
export NAMESPACE=openshift-operators

oc patch csv k8s-triliovault-stable.$VERSION --type='json'   -p='[{"op": "replace", "path": "/spec/install/spec/deployments/4/spec/template/spec/containers/0/resources/limits/memory", "value":"2048Mi"}]'

kubectl delete pod -l app=k8s-triliovault-web-backend -n $NAMESPACE

The following snippet shows how to increase the CPU value to 800M for the web-backend pod

export VERSION=2.6.0
export NAMESPACE=openshift-operators

oc patch csv k8s-triliovault-stable.$VERSION --type='json'   -p='[{"op": "replace", "path": "/spec/install/spec/deployments/4/spec/template/spec/containers/0/resources/limits/cpu", "value":"800m"}]'

kubectl delete pod -l app=k8s-triliovault-web-backend -n $NAMESPACE

After performing this workaround you will see two k8s-triliovault-web-backend pods because the replica set will be set to 2. This happens because Trilio sets therevisionhistoryto 1. Any changes to the csv and/or deployment will leave behind an extra replicaset. This can be resolved by manually deleting the extra replicaset or by setting revisionhistory to 0.

Uninstall Trilio

To uninstall Trilio and start fresh, delete all Trilio CRD's.

On Kubernetes

kubectl delete crds $(kubectl get crds | grep trilio | awk '{print $1}')

On OpenShift

oc delete crds $(kubectl get crds | grep trilio | awk '{print $1}')

Coexisting with service mesh

A service mesh assigned to a namespace can cause Trilio containers to fail to exit due to the injection of sidecar containers. Trilio operations, including the creation of targets, backups, and restores, may encounter indefinite execution or failure due to timeout.

By running the following commands, one can determine if a non-Trilio container is running inside the pod.

kubectl get pods -n NAMESPACE

kubectl describe pod PODNAME -n NAMESPACE

Trilio has the ability to coexist with a service mesh if it supports exclusion of sidecar init containers from pods based on labeling. Additionally, Trilio offers optional flags that users can provide as needed.

Edit the Trilio "tvm" CR by running the below command.

The namespace is dependent on the Kubernetes distro and Trilio version.

kubectl edit tvm -n NAMESPACE

Add or edit the section “helmvalues” as shown below for the respective service mesh. Please follow the respective service mesh software’s best practices for assigning the label. Below is an example of linkerd service mesh.

spec:
  applicationScope: Cluster
  helmValues:
    PodAnnotations:
      linkerd.io/inject: disabled

Note : Trilio for Kubernetes automatically performs this exclusion for Istio and Portshift service mesh sidecars; no further action is required. There is an enhancement around this in Kubernetes version 1.27