Unable to flow kuberbenetes logs to elasticsearch

[arunreddy00]

Hello Elastic,
Really need your help struggling to get logs to ElasticSearch

1. Created EKS cluster in AWS with two nodes
2. Later integrated VPN (site to site connection) from EKS cluster's VPC to connect to Office network.
3. Established Vpn connection to flow kuberbenetes logs from Eks to Elasticsearch which is in office network.

• Able to ping and telnet from EKS cluster nodes,but Kuberbenetes logs are not going to Elasticsearch.

4. Deployed kube-state-metrics on EKS cluster
5. later deployed metricbeat-kubernetes.yaml from below link
   https://github.com/elastic/beats/blob/7.9.3/deploy/kubernetes/metricbeat-kubernetes.yaml

•kubectl create -f metricbeat-kubernetes.yaml

changed below fields:

output.elasticsearch:
hosts: ['10.10.10.11:9200']
username: ${ELASTICSEARCH_USERNAME}
password: ${ELASTICSEARCH_PASSWORD}

env:
-name: ELASTICSEARCH_HOST
value: 10.10.10.11
- name: ELASTICSEARCH_PORT
value: "9200"
- name: ELASTICSEARCH_USERNAME
value: elastic
- name: ELASTICSEARCH_PASSWORD
value: jkjgshfaytioutgsaifug

Error logs:
please find attachment

[xeraa]

1. Is that really HTTP (as in the connection attempt) or HTTPS?
2. Since you say that you can telnet, what happens when you run a cURL request? And that is from the EKS cluster, right?

[arunreddy00]

Thank you xeraa, for response .Yes, this is from eks cluster

Figured out what went wrong.

replaced with below fields with earlier fields

output.elasticsearch:
hosts: ['${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}']
username: ${ELASTICSEARCH_USERNAME}
password: ${ELASTICSEARCH_PASSWORD}
ssl.verification_mode: none

env:
- name: ELASTICSEARCH_HOST
value: https://10.10.10.11
- name: ELASTICSEARCH_PORT
value: "9200"
- name: ELASTICSEARCH_USERNAME
value: elastic
- name: ELASTICSEARCH_PASSWORD
value: jkjgshfaytioutgsaifug

Now, Logs are visible in the ElasticSearch dashboard.

But, In Controller Manager & API server ECS,&Scheduler no metrics are shown, dashboards are empty .

I don't understand why these dashboards are empty, Can i know why does the dashboards are not filled with metrics.

[xeraa]

So what data do you have in the Metricbeat index then?

[arunreddy00]

1.In Elasticsearch dashboard , have detailed metrics of overview of kubernetes cluster metrics and kubernetes proxy metrics.

2.But Controller Manager & API server ECS,&Scheduler dashboards are empty.

• Is this happening because of EKS cluster..?

[xeraa]

I've never used EKS, so "maybe". The docs know what data and fields that should be collecting — do you get that data in your cluster?

[arunreddy00]

where do i get docs.?
To compare with..

As i didn't find any docs about EKScluster metrics dashboard's in Elasticsearch

[xeraa]

https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-metricset-kubernetes-controllermanager.html has the expected fields. In Kibana's Discover you can filter the data down to metricset.name : "controllermanager".

The first step will be to figure out if you are collecting the necessary data. Based on that the next step will be: Either look at the Metricbeat logs to see why they are not being collected; or debug the dashboard why they are not being displayed correctly.

PS: I assume you have enabled the required module, right?

- module: kubernetes
  enabled: true
  metricsets:
    - state_node
    - state_deployment
    - state_replicaset
    - state_statefulset
    - state_pod
    - state_container
    - state_cronjob
    - state_resourcequota
    - state_service
    - state_persistentvolume
    - state_persistentvolumeclaim
    - state_storageclass

[arunreddy00]

Yes, I enabled that module

For reference, i used below link to deploy

elastic/beats/blob/7.9/deploy/kubernetes/metricbeat-kubernetes.yaml

[xeraa]

Good. So can you find the data from that module in Kibana's Discover? Otherwise we'll need to take a look at the logs.

[arunreddy00]

Dear Xeraa,

After uncomment below field I see data in the dashboard. But the data is wrong which i see in the dashboard.

- module: kubernetes
metricsets:
- apiserver
hosts: ["https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}"] bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
ssl.certificate_authorities:
- /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
period: 30s

The data i see in the dashboard is , please find attachment

But in EKS cluster i only have

• node :1
• secrets : 1
• services: 1
• namespaces: 4
• configmaps: 1
• leases: 0
• endpoints: 1

Can you please look into the issue .

[xeraa]

Let's see why:

• Click "Edit" on your dashboard.
• Open the visualization (each visualization will have an icon in the top right corner that you can open and then edit it)
• On the visualization there should be an "Inspect" button that shows you the actual query and response.

With that information we can figure out, how it is getting to that result.

[arunreddy00]

When i went through the steps you mentioned, to "Inspect" dashboard, it shows Disabled.
We assume this type of visualization does not support inspect.
Please find below attachment

Thank you Xeraa, For helping.

[xeraa]

Sorry, on some visualizations this isn't available (and I always forget which ones and in which version), so we'll have to look at the configuration for that one. And actually the visualization without data would be the interesting one.
What's the index and setting a little further down on the visualization?