Not able to see latest logs in discover in kibana

Elastic Stack Kibana
1

Hi Team,

I am not able to see latest logs in discover tab in kibana. the last logs I can see is of 1 Dec 2020 but not after that.

I can see folders getting created till today in /opt/elasticsearch/nodes/0/indices/, then why new logs are not shown in kibana.

(/opt/elasticsearch is path.data)

I checked all the services are up. What I need to check to fix this issue?

Can you please help

Thanks,

2

Check date filter to ensure the correct range is correct?

If you look at the index under Management do you see the size or doc count increasing?
Is the heath yellow or green?

image
image2882×828 201 KB

How are you ingesting the data?

3

Hi @aaron-nimocks,

Thanks for your reply.

Check date filter to ensure the correct range is correct

How can I check this?

Data flow goes like,
Filebeat to kafka , kafka to logstatsh and then to
elasticsearch.

4

Check that filter and just use last 24 or 48 hours. Something after December 1st.

image
image2928×1764 560 KB

5

Hi @aaron-nimocks,

When I login to kibana and click on discover, 15 mins is by default selected and I see no logs. I have to change that to 7 days ago and then I see last logs of 1 dec.

6

Do you have Stack Monitoring setup where you can see your Logstash ingestion stats? It would show if you have been receiving data.

Or I would check the logstash logs to see if anything is being processed.

7

Hi @aaron-nimocks,

Thanks for reply. yesterday I performed some housekeeping on /var,/opt to free up some space and also kafka service was down, I restarted it and was able to see logs in filebeat.

I saw some numbers in logstash in stack monitoring.

I will monitor if new logs are coming to kibana or not.

8

I can see yesterday's (5th Dec) index got created after making some space and restarting kafka.

below is the screen shot of filebeat indices.

filebeat_indices
filebeat_indices1013×600 60.7 KB

Why it is saying 7 indices have lifecycle errors? what it means?

for how many days, indices have retention period and do they get auto deleted after that period?

below screen shot is after clicking on Show errors

lifecycle_errors

Thanks,

9

Hi @aaron-nimocks

Can you please reply.

10

I am not sure for ILM policies. Might be best to make a new topic for this specific issue.

Most likely it was an error due to the disk being full?

11

Hi @aaron-nimocks

Thanks for your quick response.

I will create new topic but its always there even if there is free space.