Not getting the latest logs in kibana

[Dev220 (Dev)]

Hi Team,

I am new to kibana.
For collecting the logs from my application pods I am using fluent-bit configuration as a pod.
Both the Elasticsearch and kiban services are up and running.
I am able to see the logs in kibana dashboard but not the latest ones.
For some pods I am seeing 1day older logs in kibana dashboard although recent logs are present inside the pods.
And for some other pods I am seeing logs in 30 minutes delay.

So my issue is I am unable to see the latest logs in Kibana dashboard.
I have tried recreating the fluent bit pod by changing the fluent bit image.
After 10 minutes of re-deploying the fluent bit pod I am still facing the same issue.

Can any body help me in this.
Thank you

[stephenb (Stephen Brown)]

Hi @Dev220 Welcome to the community.

At the macro level you may be overloading your elasticseach cluster... or perhaps you are overloading your fluent components?

But we can not know until you tell us a bit more about your cluster and the amount of logs you are shipping.

What is the size and configuration of your Elasticsearch cluster?

What kind / how much of HW / Storage / RAM / CPU?

About how many logs / minute are you shipping? What kind of logs?

How many Pods and fluent-bits (although we are not experts on that?)

Is it straight from fluent-bit -> Elasticsearch or are there other components in the middile?

[Dev220 (Dev)]

Hi stephenb

Please find below details which you have asked for.

Fluent bit we are running as a container on our EKS cluster

What is the size and configuration of your Elasticsearch cluster?
There is no cluster for Elasticsearch we are maintaining our Elasticsearch on a standalone server.

What kind / how much of HW / Storage / RAM / CPU?
0.5 CPU and 1GB of RAM for fluent bit

About how many logs / minute are you shipping? What kind of logs?
Around 1GB logs /day we are shipping and application logs (text logs)

How many Pods and fluent-bits (although we are not experts on that?)
2 environments and in each pod 17 pods
apart from that 1 pod fluent bit in each env

Is it straight from fluent-bit -> Elasticsearch or are there other components in the middile?
Yes it is straight from fluent bit to Elasticsearch
No middle components are there

Thanks

[stephenb (Stephen Brown)]

@Dev220

There is no cluster for Elasticsearch we are maintaining our Elasticsearch on a standalone server.

Is it a single node? I wanted to know the specs on elasticsearch.. If elasticsearch is not big enough, it could be backing up.

If elasticsearch is standalone have you looked at the logs?

Is that 1 GB a day for each pod or a total shipped from all pods?

[Dev220 (Dev)]

I wanted to know the specs on Elasticsearch.
Ans-4 CPU , 16GB memory

If Elasticsearch is standalone have you looked at the logs?
I am not sure what is the path for seeing the logs for Elasticsearch

Is that 1 GB a day for each pod or a total shipped from all pods?
Ans - Total shipped logs from all the pods is around 1.8GB-2GB

[stephenb (Stephen Brown)]

Looks like you should have sufficient resources... but it is hard to tell not sure if that is a Single node or multiple, if Single still should be fine.

I am still unclear where / how Elasticsearch is deployed... is it in the same k8s Cluster?
If in K8s you should be able to log into the container and find the logs
If on a host should be /var/log/elasticsearch or journalctl

I am not that familiar with fluent so if there is a problem there I am not the right one to answer.

You going to need to dig in and provide some additional detail so we can help... otherwise we are just guessing.

You could try filebeat