Monitoring T4K Logs using ELK Stack
This section provides step by step instructions to install and configure Elastic Search, Logstash and Kibana to monitor T4K logs with other native Kubernetes resources.
In Kubernetes, its important for users to be able to visualize the logging from different applications running in the environment. ELK stack can be used and integrated with Trilio For Kubernetes to read, parse and visualize the logs on a dashboard.
ELK stack consist of Elastic Search, Logstash and Kibana.
- Elastic Search parses and stores the logs received from Logstash
- Logstash receives the data from Beats (Agent who reads the data from logfiles for forward to Logstash) and parse it
- Kibana is a dashboard to visualize the logs stored in Elastic database and run the queries on this data to see the desired results
- 1.Install Custom Resource Definitions
kubectl create -f https://download.elastic.co/downloads/eck/1.7.1/crds.yaml
2. Verify the CRDs are deployed correctly on the Kubernetes cluster
kubectl get crd | grep elastic
agents.agent.k8s.elastic.co 2021-08-30T07:36:15Z
apmservers.apm.k8s.elastic.co 2021-08-30T07:36:15Z
beats.beat.k8s.elastic.co 2021-08-30T07:36:16Z
elasticmapsservers.maps.k8s.elastic.co 2021-08-30T07:36:16Z
elasticsearches.elasticsearch.k8s.elastic.co 2021-08-30T07:36:16Z
enterprisesearches.enterprisesearch.k8s.elastic.co 2021-08-30T07:36:16Z
kibanas.kibana.k8s.elastic.co 2021-08-30T07:36:16Z
3. Install Operator using yaml definition
kubectl apply -f https://download.elastic.co/downloads/eck/1.7.1/operator.yaml
4. Verify that the
elastic-system namespace is created and elastic-operator statefulset is deployed. The elastic-operator-0 pod must be in Running statekubectl get all -n elastic-system
NAME READY STATUS RESTARTS AGE
pod/elastic-operator-0 1/1 Running 0 1m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/elastic-webhook-server ClusterIP 10.60.6.150 <none> 443/TCP 1m
NAME READY AGE
statefulset.apps/elastic-operator 1/1 1m
- 1.Create an elasticsearch cluster definition yaml file
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: quickstart
spec:
version: 7.14.0
nodeSets:
- name: default
count: 2
config:
node.store.allow_mmap: false
Note: User can change
spec.nodeSets[0].count to 1 or more than 1. This parameter defines the number of pods deployed in the Elasticsearch cluster.If your Kubernetes cluster does not have at least 2GB of free memory then the pod will be stuck in
Pending state.2. Apply the elasticsearch definition yaml file
kubectl apply -f elasticsearch.yaml
3. Once the pods are deployed, it may take a few minutes until all the resources are created and the elasticsearch cluster is ready to use
4. Monitor the cluster health and creation progress
kubectl get elasticsearch
NAME HEALTH NODES VERSION PHASE AGE
quickstart green 2 7.14.0 Ready 1m
5. You can see the two pods deployed and in
Running statekubectl get pods --selector='elasticsearch.k8s.elastic.co/cluster-name=quickstart'
NAME READY STATUS RESTARTS AGE
quickstart-es-default-0 1/1 Running 0 1m
quickstart-es-default-1 1/1 Running 0 1m
6. Get the credentials.
A default user names elastic is automatically create with the password stored in Kubernetes secret
PASSWORD=$(kubectl get secret quickstart-es-elastic-user -o go-template='{{.data.elastic | base64decode}}')
echo $PASSWORD
Note: Save the $PASSWORD value since you will need this for login to elasticsearch cluster
7. Request the Elasticsearch endpoint from inside the Kubernetes cluster.
curl -u "elastic:$PASSWORD" -k "https://quickstart-es-http:9200"
8. From your local workstation, run
port-forward command from a different terminalkubectl port-forward service/quickstart-es-http 9200
curl -u "elastic:$PASSWORD" -k "https://localhost:9200"
{
"name" : "quickstart-es-default-1",
"cluster_name" : "quickstart",
"cluster_uuid" : "tkOvQq_5SgCEkTyzxmJqFQ",
"version" : {
"number" : "7.14.0",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "dd5a0a2acaa2045ff9624f3729fc8a6f40835aa1",
"build_date" : "2021-07-29T20:49:32.864135063Z",
"build_snapshot" : false,
"lucene_version" : "8.9.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
- 1.Create an elasticsearch cluster definition yaml file
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
name: quickstart
spec:
version: 7.14.0
count: 1
elasticsearchRef:
name: quickstart
Note: User can change
spec.count from 1 to more than 1. This parameter defines the number of pods deployed for the Kibana dashboard.2. Apply the Kibana dashboard definition yaml file
kubectl apply -f kibana.yaml
3. Once the pods are deployed, it may take a few minutes until all the resources are created and the Kibana dashboard is ready to use
4. Monitor the Kibana health and creation progress
kubectl get kibana
NAME HEALTH NODES VERSION AGE
quickstart green 1 7.14.0 1m
5. You can see the Kibana quickstart pod deployed and in
Running statekubectl get pod --selector='kibana.k8s.elastic.co/name=quickstart'
NAME READY STATUS RESTARTS AGE
quickstart-kb-7966b84d57-rzcf2 1/1 Running 0 1m
6. Access Kibana dasboard.
A
ClusterIP service is automatically created fro Kibanakubectl get service quickstart-kb-http
From your local workstation, run
port-forward command from a different terminalkubectl port-forward service/quickstart-kb-http 5601
7. Access the Kibana Dashboard from your workstation browser by navigating to below URL
https://localhost:5601
8. Use the username as
elastic and password generated in elasticsearch cluster deployment
Kibana Dashboard Login Page