# Known Issues and Workarounds

### Management Console not accessible using NodePort

If user has done all the configuration mentioned as per the [Accessing the Management Console](https://docs.trilio.io/kubernetes/advanced-configuration/management-console/accessing-the-ui#set-up-access-via-port-forwarding) instructions and still not able to access the UI, check the Firewall Rules for the Kubernetes cluster nodes.

Here is an example to check firewall rules on the Google GKE cluster:

1. Search the VPC Network on GCP web console
2. Select \`Firewall\` option from the left pane

![Select Firewall from VPC Network](https://content.gitbook.com/content/9sDjF5HJP1bf8TtLcgkk/blobs/Xl5If71uDUthGFM4lOHL/image%20\(8\).png)

1. Search the rules with your Kubernetes cluster name
2. In the list of firewalls, verify:
   1. **Filters** column - Shows the source IP which can access the T4K Management Console Hostname and `NodePort`
   2. **Protocols/Ports** column - Shows the ports which are accessible on the cluster nodes.
3. Verify that the `NodePort` assigned to service `k8s-triliovault-ingress-gateway` is added in the firewall rules Protocols and Ports.

![Filters showing IP range and Protols/Ports showing exposed Port](https://content.gitbook.com/content/9sDjF5HJP1bf8TtLcgkk/blobs/9gHRgv67Ki5CAZL8jzXV/image%20\(125\).png)

### Target Creation Failure

**`kubectl apply -f <backup_target.yaml>`** is not successful and the resource goes into `Unavailable` state.

Possible reasons for the target marked `Unavailable` could be not having enough permissions or lack of network connectivity. Target resource creation launches a validation pod to check if Trilio can connect and write to the Target. Trilio then cleans the pod after the validation process completes, which clears any logs that the validation pod has. Users should repeat the same operation and actively check the logs on the validation pods.

#### Kubernetes

```
kubectl get pods -A | grep validator
kubectl logs <validator-pod> -n <namespace>
```

#### OpenShift

```
oc get pods -A | grep validator
oc logs <validator-pod> -n <namespace
```

### Backup Failures

Please follow the following steps to analyze backup failures.

```
kubectl get jobs
oc get jobs
```

To troubleshoot backup issues, it is important to know the different phases of `jobs.triliovault` pod.

1. `<backupname>-snapshot`: Trilio leverages CSI snapshot functionality to take PV snapshots. If the backup fails in this step user should try a manual snapshot of the PV to make sure all required drivers are present, and that this operation works without Trilio.
2. `<backupname>-upload`: In this phase, Trilio uploads metadata and data to the Target. If the backup fails in this phase, we need to look into the following logs for error.

#### Kubernetes

```
kubectl get pods -A | grep k8s-triliovault-control-plane
kubectl logs <controller POD>
kubectl logs <failed or errored datamover pod>
```

#### OpenShift

```
oc get pods -A | grep k8s-triliovault-control-plane
oc logs <controller POD>
oc logs <failed or errored datamover pod>
```

### Restore Failures

The most common reasons for restore failures are resource conflicts. Users may try to restore the backup to the same namespace where the original application and restored application have names conflict. Please check the logs to analyze the root cause of the issue.

#### Kubernetes

```
kubectl get pods -A | grep k8s-triliovault-control-plane
kubectl logs <controller POD>
kubectl logs <failed or errored metamoved pod>
```

#### OpenShift

```
oc get pods -A | grep k8s-triliovault-control-plane
oc logs <controller POD>
oc logs <failed or errored metamoved pod>
```

#### Webhooks are installed

*Problem Description*: If a Trilio operation is failing with an error `Internal error occurred: failed` or `service "k8s-triliovault-webhook-service" not found`

```
Error from server (InternalError): error when creating "targetmigration.yaml": Internal error occurred: failed 
calling webhook "mapplication.kb.io": Post https://k8s-triliovault-webhook-service.default.svc:443/mutate?timeout=30s: service "k8s-triliovault-webhook-service" not found
```

*Solution* : This might be due to the installation of T4K in multiple Namespaces. Even a simple clean-up of those installations will not have webhook instances cleared. Please run the following commands to get all the webhooks instances.

```
kubectl get validatingwebhookconfigurations  -A | grep trilio
kubectl get mutatingwebhookconfigurations -A | grep trilio
```

Delete duplicate entries if any.

#### Restore Failed

*Problem Description*: If you are not able to restore an application into different namespace (OpenShift)

```
Error:
Permission denied: could not bind to address 0.0.0.0:80
```

*Solution*: User may not have sufficient permissions to restore the application into the namespace.

```
oc adm policy add-scc-to-user anyuid -z default -n <NameSpace>O
```

### OOMKilled Issues

Sometimes users may notice some Trilio pods go into OOMKilled state. This can happen in environments where a lot of objects exist for which additional processing power may be needed. In such scenarios, the memory and CPU limits can be bumped up by running the following commands:

#### Kubernetes

Please follow the [API documentation](https://docs.trilio.io/kubernetes/reference-guides/custom-resource-definition-operator#user-content-triliovault.trilio.io-v1.componentconfiguration) to increase resource limits for Trilio pods deployed via the upstream Helm based Operator

#### OCP

Within OCP, the memory and CPU values for Trilio are managed and controlled by the Operator `ClusterServiceVersion(CSV)`Users can either edit the Operator CSV YAML directly from within the OCP console or can patch the pods based on the snippets provided below.

The following snippet shows how to increase the memory value to 2048Mi for the `web-backend` pod

```
export VERSION=2.6.0
export NAMESPACE=openshift-operators

oc patch csv k8s-triliovault-stable.$VERSION --type='json'   -p='[{"op": "replace", "path": "/spec/install/spec/deployments/4/spec/template/spec/containers/0/resources/limits/memory", "value":"2048Mi"}]'

kubectl delete pod -l app=k8s-triliovault-web-backend -n $NAMESPACE
```

The following snippet shows how to increase the CPU value to 800M for the `web-backend` pod

```
export VERSION=2.6.0
export NAMESPACE=openshift-operators

oc patch csv k8s-triliovault-stable.$VERSION --type='json'   -p='[{"op": "replace", "path": "/spec/install/spec/deployments/4/spec/template/spec/containers/0/resources/limits/cpu", "value":"800m"}]'

kubectl delete pod -l app=k8s-triliovault-web-backend -n $NAMESPACE
```

{% hint style="warning" %}
After performing this workaround you will see two k8s-triliovault-web-backend pods because the replica set will be set to 2. This happens because Trilio sets the`revisionhistory`to 1. Any changes to the csv and/or deployment will leave behind an extra replicaset. This can be resolved by manually deleting the extra replicaset or by setting `revisionhistory` to 0.
{% endhint %}

### Uninstall Trilio

To uninstall Trilio and start fresh, delete all Trilio CRD's.

On Kubernetes

```
kubectl delete crds $(kubectl get crds | grep trilio | awk '{print $1}')
```

On OpenShift

```
oc delete crds $(kubectl get crds | grep trilio | awk '{print $1}')
```

### **Coexisting with service mesh**

A service mesh assigned to a namespace can cause Trilio containers to fail to exit due to the injection of sidecar containers. Trilio operations, including the creation of targets, backups, and restores, may encounter indefinite execution or failure due to timeout.

By running the following commands, one can determine if a non-Trilio container is running inside the pod.

`kubectl get pods -n NAMESPACE`

`kubectl describe pod PODNAME -n NAMESPACE`

Trilio has the ability to coexist with a service mesh if it supports exclusion of sidecar init containers from pods based on labeling. Additionally, Trilio offers optional flags that users can provide as needed.

Edit the Trilio "tvm" CR by running the below command.

The namespace is dependent on the Kubernetes distro and Trilio version.

`kubectl edit tvm -n NAMESPACE`

Add or edit the section “helmvalues” as shown below for the respective service mesh. Please follow the respective service mesh software’s best practices for assigning the label. Below is an example of **linkerd** service mesh.

```yaml
spec:
  applicationScope: Cluster
  helmValues:
    PodAnnotations:
      linkerd.io/inject: disabled
```

{% hint style="info" %}
Note : Trilio for Kubernetes automatically performs this exclusion for Istio and Portshift service mesh sidecars; no further action is required. There is an enhancement around this in Kubernetes version 1.27
{% endhint %}