Kibana not updating index in Discover

[zaeemmasood]

Hi All,

I see this issue where Kibana is not updating index on the "Discover" page while there is a definite increase in the size of the related index. Also for some reason Discover page shows data with one hour interval. Please see below:

The "Storage Size" keeps incrementing with no corresponding display on the "Discover" page

Health of the hosting servers in terms of CPU and memory is fairly ok.

Please guide.

[stephenb (Stephen Brown)]

Hi @zaeemmasood

There are a couple of possible reasons this could happen.

What is the refresh interval set to? The docs will not show until the refresh interval is past.

More likely what is happening is that the incoming documents are outside your time selection... i.e. you have some old logs coming in that are before your time window (or the are coming in ahead if you have a timezone miss-match) ... so open up / widen the time frame and see if it aligns.

[zaeemmasood]

Thanks.

I am doing manual refresh after setting interval as last 5 hours.

Even if I set it to last 15 minutes it doesnt show any data while the index size increases gradually.

I know for sure that the data is continuously coming in.

[zaeemmasood]

Also a very strange thing was noted. I see that the index size "Storage size" gets reduced now. Notice it was 123.1 gb earlier

[stephenb (Stephen Brown)]

@zaeemmasood

I am doing manual refresh after setting interval as last 5 hours.

I am not talking about Refresh in Kibana .... I am talking about the refresh interval at the index level but that is probably not the issue...

In short the most likely logs that are coming in ... are not within the time window you are looking at.

So try this set the time interval to the following

Last 7 Days (or 14 Days) and 1 Day in the future.

[stephenb (Stephen Brown)]

Storage can get reduced when a merges (compaction) happens... the doc count looks like it is increasing

[zaeemmasood]

Thanks!

Looks like after I set the date range as per what you said I see continuous stream of messages. See below

Question is why is it showing time of 11:35AM EST when I refresh just now? Is there a lag somewhere?

[stephenb (Stephen Brown)]

That time stamp you highlighted is the time stamp converted to the Time Zone in the browser OR perhaps you changed a setting in Kibana Advanced Setting...

Discover show relative to the Kibana Setting .. You can see the actual UTC in the document

Could be lag
Could be time zone issue
Could be index refresh interval
Combination...
Something else

With out detailed debugging hard to know...

[zaeemmasood]

thanks @stephenb

This set up has been running fine for over 2 years now and this issue has cropped up all of sudden. Also browser setting in terms of Kibana has not changed ever since.

Index refresh interval is set to default 1s.

Servers health has been also showing ok.

That leaves "lag" which could be because of some "network issue".

Please let me know if you agree?

A lot of following errors are also seen in the filebeat log:

2023-04-27T14:38:20.986+0100    ERROR   pipeline/output.go:121  Failed to publish events: write tcp 19.18.28.25:40806->14.18.8.8:5044: write: connection reset by peer
2023-04-27T14:38:21.059+0100    ERROR   pipeline/output.go:121  Failed to publish events: write tcp 16.18.28.45:47966->14.12.7.6:5044: write: connection reset by peer

Please note that the logstash instance listening on port 5044 consumes more than one feed from various application logs.

[stephenb (Stephen Brown)]

So Lag is most likely the case...

Lag AKA Backpressure can come from many reason or combinations of reasons and can be hard to diagnose... you are going to need to look closely at every step...

@zaeemmasood

That leaves "lag" which could be because of some "network issue".

I am not sure I would immediately jump to that... but that certainly could be a reason... I have seen LBs or switches get saturated.

With those error messages it is possible the logstash is getting overwhelmed.

Is there new logs volume being sent to logstash?

You just going to need to dig in... no easy way out

[zaeemmasood]

Thanks. There has not been any change lately in terms of logs.

Logstash instance listening on port 5044 has around 20 applications pushing logs from various servers and indices getting created. Interestingly, the issue is in only 1 out of the 20 applications. Rest all the 19 apps have their indices getting updated without any delay. The only difference is that 1 application creates the most traffic and has 24 application servers.

[stephenb (Stephen Brown)]

@zaeemmasood

The only difference is that 1 application creates the most traffic and has 24 application servers.

Perhaps the Shipping side is not keeping up for that app