# Monitoring T4K Logs using ELK Stack ## Introduction In Kubernetes, its important for users to be able to visualize the logging from different applications running in the environment. ELK stack can be used and integrated with Trilio For Kubernetes to read, parse and visualize the logs on a dashboard. ### What is ELK Stack ELK stack consist of Elastic Search, Logstash and Kibana. * Elastic Search parses and stores the logs received from Logstash * Logstash receives the data from Beats (Agent who reads the data from logfiles for forward to Logstash) and parse it * Kibana is a dashboard to visualize the logs stored in Elastic database and run the queries on this data to see the desired results ### Install and configure ELK on Kubernetes #### Install CRDs and Operators 1. Install Custom Resource Definitions ``` kubectl create -f https://download.elastic.co/downloads/eck/1.7.1/crds.yaml ``` 2\. Verify the CRDs are deployed correctly on the Kubernetes cluster ```bash kubectl get crd | grep elastic ``` ```bash agents.agent.k8s.elastic.co 2021-08-30T07:36:15Z apmservers.apm.k8s.elastic.co 2021-08-30T07:36:15Z beats.beat.k8s.elastic.co 2021-08-30T07:36:16Z elasticmapsservers.maps.k8s.elastic.co 2021-08-30T07:36:16Z elasticsearches.elasticsearch.k8s.elastic.co 2021-08-30T07:36:16Z enterprisesearches.enterprisesearch.k8s.elastic.co 2021-08-30T07:36:16Z kibanas.kibana.k8s.elastic.co 2021-08-30T07:36:16Z ``` 3\. Install Operator using yaml definition ```bash kubectl apply -f https://download.elastic.co/downloads/eck/1.7.1/operator.yaml ``` 4\. Verify that the `elastic-system` namespace is created and `elastic-operator` statefulset is deployed. The `elastic-operator-0` pod must be in `Running` state ```bash kubectl get all -n elastic-system ``` ```bash NAME READY STATUS RESTARTS AGE pod/elastic-operator-0 1/1 Running 0 1m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/elastic-webhook-server ClusterIP 10.60.6.150 443/TCP 1m NAME READY AGE statefulset.apps/elastic-operator 1/1 1m ``` #### Deploy an Elasticsearch Cluster 1. Create an elasticsearch cluster definition yaml file ```bash apiVersion: elasticsearch.k8s.elastic.co/v1 kind: Elasticsearch metadata: name: quickstart spec: version: 7.14.0 nodeSets: - name: default count: 2 config: node.store.allow_mmap: false ``` {% hint style="info" %} **Note:** User can change `spec.nodeSets[0].count` to 1 or more than 1. This parameter defines the number of pods deployed in the Elasticsearch cluster. If your Kubernetes cluster does not have at least 2GB of free memory then the pod will be stuck in `Pending` state. {% endhint %} 2\. Apply the elasticsearch definition yaml file ```bash kubectl apply -f elasticsearch.yaml ``` 3\. Once the pods are deployed, it may take a few minutes until all the resources are created and the elasticsearch cluster is ready to use 4\. Monitor the cluster health and creation progress ```bash kubectl get elasticsearch ``` ```bash NAME HEALTH NODES VERSION PHASE AGE quickstart green 2 7.14.0 Ready 1m ``` 5\. You can see the two pods deployed and in `Running` state ```bash kubectl get pods --selector='elasticsearch.k8s.elastic.co/cluster-name=quickstart' ``` ```bash NAME READY STATUS RESTARTS AGE quickstart-es-default-0 1/1 Running 0 1m quickstart-es-default-1 1/1 Running 0 1m ``` 6\. Get the credentials. A default user names elastic is automatically create with the password stored in Kubernetes secret ```bash PASSWORD=$(kubectl get secret quickstart-es-elastic-user -o go-template='{{.data.elastic | base64decode}}') ``` ```bash echo $PASSWORD ``` {% hint style="info" %} **Note:** Save the $PASSWORD value since you will need this for login to elasticsearch cluster {% endhint %} 7\. Request the Elasticsearch endpoint from inside the Kubernetes cluster. ```bash curl -u "elastic:$PASSWORD" -k "https://quickstart-es-http:9200" ``` 8\. From your local workstation, run `port-forward` command from a different terminal ```bash kubectl port-forward service/quickstart-es-http 9200 ``` ```bash curl -u "elastic:$PASSWORD" -k "https://localhost:9200" ``` ```bash { "name" : "quickstart-es-default-1", "cluster_name" : "quickstart", "cluster_uuid" : "tkOvQq_5SgCEkTyzxmJqFQ", "version" : { "number" : "7.14.0", "build_flavor" : "default", "build_type" : "docker", "build_hash" : "dd5a0a2acaa2045ff9624f3729fc8a6f40835aa1", "build_date" : "2021-07-29T20:49:32.864135063Z", "build_snapshot" : false, "lucene_version" : "8.9.0", "minimum_wire_compatibility_version" : "6.8.0", "minimum_index_compatibility_version" : "6.0.0-beta1" }, "tagline" : "You Know, for Search" } ``` #### Install Kibana Dashboard 1. Create an elasticsearch cluster definition yaml file ```bash apiVersion: kibana.k8s.elastic.co/v1 kind: Kibana metadata: name: quickstart spec: version: 7.14.0 count: 1 elasticsearchRef: name: quickstart ``` {% hint style="info" %} **Note:** User can change `spec.count` from 1 to more than 1. This parameter defines the number of pods deployed for the Kibana dashboard. {% endhint %} 2\. Apply the Kibana dashboard definition yaml file ```bash kubectl apply -f kibana.yaml ``` 3\. Once the pods are deployed, it may take a few minutes until all the resources are created and the Kibana dashboard is ready to use 4\. Monitor the Kibana health and creation progress ```bash kubectl get kibana ``` ```bash NAME HEALTH NODES VERSION AGE quickstart green 1 7.14.0 1m ``` 5\. You can see the Kibana quickstart pod deployed and in `Running` state ```bash kubectl get pod --selector='kibana.k8s.elastic.co/name=quickstart' ``` ```bash NAME READY STATUS RESTARTS AGE quickstart-kb-7966b84d57-rzcf2 1/1 Running 0 1m ``` 6\. Access Kibana dasboard. A `ClusterIP` service is automatically created fro Kibana ```bash kubectl get service quickstart-kb-http ``` From your local workstation, run `port-forward` command from a different terminal ```bash kubectl port-forward service/quickstart-kb-http 5601 ``` 7\. Access the Kibana Dashboard from your workstation browser by navigating to below URL ```bash https://localhost:5601 ``` 8\. Use the username as `elastic` and password generated in elasticsearch cluster deployment ![Kibana Dashboard Login Page](https://content.gitbook.com/content/9sDjF5HJP1bf8TtLcgkk/blobs/CBHayXzHfDg2s5hN5GLa/Kibana%20dashboard.png)