Time filter not working in kibaba 7.4 when any one of the shard is down

[Sundar.rv (Sundararaman)]

Hi everyone,

Currently we are testing our ELK cluster where we have 5 data nodes. Our index are split up in 5 data nodes (5 shards) as the index size is ~45GB.

The index is a live one (its getting events from logstash continuously), when one of the node when down, we observed the time filter in Kibana is not fully functional.

1. If I choose the time filter as Today, it shows me the data all the latest events, but when I give last 15 mins, it wasn’t showing me anything. This is quite strange..even after 30 mins I tried, it wasn't showing me any data. But the number of docs in the index kept increasing.

2. Applying any search with time filter as Today is working, but when I tried with 15 or 30 mins, it doesn't give me any data (it should bring results as it was showing data with Today as time filter).

Any idea on this behavior will help us to fine tune our setup.

Thanks,
Sundar

[flash1293 (Joe Reuter)]

I can think of two things that could cause this:

• Time zone issues - make sure the timestamp in the documents has the value you expect and not an hour shifted or something like that. This can happen for example if a date in a specific timezone is ingested, but interpreted as UTC by Elasticsearch
• Check the refresh_interval setting of your index, it might be too high. Documents don't become searchable right away but are only flushed periodically.

[Sundar.rv (Sundararaman)]

Hi Joe, Thanks for your inputs.

I tried changing the time zone to what we have in logs but still same result.

I checked the refresh_internal, it is kept at 5secs. So my guess is at least it should get me the events in UI after 10-15 secs. But still it wasn't.

Also on the time filter when I keep it as Today, the latest events in the index gets displayed. So not sure if its really due to Time zone differences or the refresh_interval settings.