I set @timezone without tailing Z and expected to see it in the Kibana in the exact shape. But I see it is timezone shifted there which I am not sure why. Putting Z or not at the end of the timestamp does not make a difference in my case.
The problem with timezone was in my test method not the original scenario. The way I saw delay in the first place is that I set a time span in search. For example from hour 0 to 'one hour ago' and queried. Recorded number of hits. After a while just clicked search button and saw the number of hits is increased. The cause of this still is unknown to me. Of course has nothing to do with timezone in this case as I didn't change the timespan for the second query. The later hit count is in accord with a related table in my database so I can validate it.
Now I am not sure if I am missing something in general or about how ES works so for now this issue may rest until I gather more info.
I set a time span in search. For example from hour 0 to 'one hour ago' and queried. Recorded number of hits. After a while just clicked search button and saw the number of hits is increased.
I hesitate to point it out, but a query from a fixed time to "one hour ago" isn't querying a constant time window, it depends on when you click the search button in kibana. Each time you click search, it re-calculates when "one hour ago" was. In that case, I'd expect the number of hits to increase, and it doesn't mean there was any delay
Adding an ingest_timestamp is a good idea in general, so please try to do that, there's a bunch threads on here showing how to do so.
Many of us have pointed out already there is no significant delay in elasticsearch making successfully ingested documents available for search, unless you specifically change it, which you say you have not, so most likely explanation here is some gap in understanding of your overall solution.
You've also been slightly opaque and vague and a bit inconsistent in the problem description, and a side alley on your test environment led to more confusion around timestamps and timezones that you assure us are not the real issue. Therefore I think you need learn some lessons from that troubleshooting process itself too, dripping limited info into the thread a bit at a time with only a portion of the information makes it really difficult to help.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.