Documents not showing up in Discover, but are showing up in queries in Dev Tools

[Arjun_Meena]

I have an index spoms-audit-2022.08.04 in which there are 4 documents. When I search them from the dev console, i am able to see all the documents.
Below is the search query:

These are the results I get back:

As we can see that I have 4 documents in total in this index pattern.

When I go to the discover tab and then in the index pattern spoms-audit*, I am not able to see any logs.

The time range I have set in Discover tab far exceeds the time the logs were generated. Yet I am not able to see any logs here..

Any help would be appreciated. Thanks!

[stephenb (Stephen Brown)]

Hi @Arjun_Meena Welcome to the Commuity.

Did you set the @timestamp field as the time field in the index pattern when you set it up?

What version are you on?

[Arjun_Meena]

Yes I did.

Here is the screenshot

[stephenb (Stephen Brown)]

What version are you on?

Do a Shift Reload of the Discover page.

[Arjun_Meena]

Logstash is 8.3 and Elasticsearch is 8.2

[Arjun_Meena]

I did Shift Reload, but still see nothing

[stephenb (Stephen Brown)]

What timezone are you in?

Try this set discover to a future time say a day in the future

Pretty sure you are loading your time with out timezone so it is in the future.

Also when you show text please paste as text not screen shot... some people can not see the images and the can not be searched or debug... example I could not cut-paste your date

The Future

[stephenb (Stephen Brown)]

Simple / Most Likely your logs are in local / IST but when you are parsing with logstash you are not setting / accounting for the IST timezone.

If no timezone is provided then Logastash / Elastic assume UTC... that is what is happening you are "Ahead" of UCT

Where ever you are processing the date on ingest you need to account for timezone.

[Arjun_Meena]

I'm updating the timestamp using the logstash filter, Is there any way I can tell it this time is in IST?

@stephenb I figured out the setting that I need to change... Thanks a lot!! We were stuck on this for about a week now..Thanks a lot for your help. You rock

[stephenb (Stephen Brown)]

@Arjun_Meena
OK good community is to not post screen shot... and if you do have a solution you should post that for others to benefit from.. you say you solved it but did not show.. please do so the next person can learn from you!

[Arjun_Meena]

The timestamp of my logs was in the local timezone (IST). I was picking the timestamp from my logs and overwriting it to the @timestamp field using logstash date filter plugin. I didn't specify any timezone setting there, so elasticsearch is taking that @timestamp as UTC and converting it to our local time zone, making them 5:30 hours ahead of their actual time.
So a log with an original timestamp of August 4th, 2022 20:40:58.364 (IST) would be shown in the discover tab as August 5th, 2022 02:10:58.364.
All I had to do to fix this was basically specify the timezone of the logs in the date filter plugin, so the elasticsearch would know that the logs timstamp are already in the IST timezone, and it doesn't need to add any offset.

Below is the working logstash config:

  #Update the timestamp with the timestamp present in the log
    date {
        match => ["timestamp", "dd-MM-yyyy'T'HH:mm:ss.SSS"]
        target => "@timestamp"
        timezone => "Asia/Kolkata"
    }