# Latency in seeing a document **URL:** \ **Category:** Elasticsearch\ **Created:** [July 13, 2024, 10:58am UTC](https://discuss.elastic.co/t/latency-in-seeing-a-document/363071 "2024-07-13T10:58:14Z")\ **Posts on this page:** 20\ **Page:** 2
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 17, 2024, 8:36am UTC](https://discuss.elastic.co/t/latency-in-seeing-a-document/363071/21 "2024-07-17T08:36:19Z")
> [@siakc](#): > > I can immediately see the doc I insert in one of the indices. For some reason I can't see it in Discover tab in Kibana. Is there a buffer of something between Kibana and ES? No, there is not. The Discover tab by default applies a timestamp filter. Is the document you inserted within this (accounting for timezone differences)? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [July 17, 2024, 8:36am UTC](https://discuss.elastic.co/t/latency-in-seeing-a-document/363071/22 "2024-07-17T08:36:44Z")
A GET by ID is realtime. A Search is not. You need to refresh or wait for the refresh to see your documents within a search request. Which is what discover is running. ---
**Author:** ![siakc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/siakc/32/135480_2.png) [@siakc](https://discuss.elastic.co/u/siakc)\ **Post date:** [July 17, 2024, 8:45am UTC](https://discuss.elastic.co/t/latency-in-seeing-a-document/363071/23 "2024-07-17T08:45:37Z")
I have "refresh\_interval": "1s" But it is about an hour and can't see a few docs I inserted. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [July 17, 2024, 9:01am UTC](https://discuss.elastic.co/t/latency-in-seeing-a-document/363071/24 "2024-07-17T09:01:25Z")
Could you share a screenshot please? ---
**Author:** ![siakc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/siakc/32/135480_2.png) [@siakc](https://discuss.elastic.co/u/siakc)\ **Post date:** [July 17, 2024, 11:01am UTC](https://discuss.elastic.co/t/latency-in-seeing-a-document/363071/25 "2024-07-17T11:01:20Z")
![image](https://us1.discourse-cdn.com/elastic/original/3X/7/1/717e6262c94f56aa65878c690c25bd43a2ebb989.png) ```auto GET /_search { "query": { "ids" : { "values" : ["g12"] } } } ``` ```auto { "took" : 317, "timed_out" : false, "_shards" : { "total" : 244, "successful" : 244, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 1, "relation" : "eq" }, "max_score" : 1.0, "hits" : [ { "_index" : "myindex-v2-2024.07.17", "_type" : "logs", "_id" : "g12", "_score" : 1.0, "_source" : { "log" : { "sof" : 334, "dd" : { "as" : "asdfjhg aslkf gaskf g", "l" : 3939 } } } } ] } } ``` ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [July 17, 2024, 11:26am UTC](https://discuss.elastic.co/t/latency-in-seeing-a-document/363071/26 "2024-07-17T11:26:29Z")
I can't see the index name but this document is only available in index myindex-v2-2024.07.17. If you need more help, please share the data view configuration. ---
**Author:** ![siakc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/siakc/32/135480_2.png) [@siakc](https://discuss.elastic.co/u/siakc)\ **Post date:** [July 17, 2024, 11:27am UTC](https://discuss.elastic.co/t/latency-in-seeing-a-document/363071/27 "2024-07-17T11:27:37Z")
> [@dadoonet](#): > > myindex Pattern is myindex\* What is data view configuration? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [July 17, 2024, 1:21pm UTC](https://discuss.elastic.co/t/latency-in-seeing-a-document/363071/28 "2024-07-17T13:21:00Z")
Note that the query you are running on Kibana \_id:g12 is not the same query as you are running within dev tools. Could you share a screen capture of the index pattern configuration? Also what is your version? ---
**Author:** ![siakc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/siakc/32/135480_2.png) [@siakc](https://discuss.elastic.co/u/siakc)\ **Post date:** [July 17, 2024, 3:32pm UTC](https://discuss.elastic.co/t/latency-in-seeing-a-document/363071/29 "2024-07-17T15:32:02Z")
Version is 7.17. Not sure where index pattern configuration is. There is Kibana\>Index Patterns which have myindex\* in it. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 17, 2024, 4:25pm UTC](https://discuss.elastic.co/t/latency-in-seeing-a-document/363071/30 "2024-07-17T16:25:12Z")
Your test document does not have the `@timestamp` field set and since your index pattern is created for data having this field present and apply a filter on it (top right corner) this test document will never show up in the results. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [July 18, 2024, 9:03am UTC](https://discuss.elastic.co/t/latency-in-seeing-a-document/363071/31 "2024-07-18T09:03:37Z")
> [@siakc](#): > > There is Kibana\>Index Patterns which have myindex\* in it. Yes. This is the screen capture I wanted to see. ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [July 19, 2024, 3:29pm UTC](https://discuss.elastic.co/t/latency-in-seeing-a-document/363071/32 "2024-07-19T15:29:02Z")
... and I fear we are now on a different issue than the one first reported. IF the original issue was with documents missing a timestamp field, they'd NEVER appear in a kibana search on the relevant index, never mind after a few seconds/minutes/hours. [siakc](https://discuss.elastic.co/u/siakc) - Data Views were introduced in v8. v7.17 is 2+ years old, not itself an issue. But as I wrote above this thread has all the hallmarks of a lack of understanding issue. Anyways, any fake / dummy document needs a timestamp, on assumption your real data also has a timestamp - see my silly insert-a-doc script above, be careful about formats. ---
**Author:** ![siakc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/siakc/32/135480_2.png) [@siakc](https://discuss.elastic.co/u/siakc)\ **Post date:** [July 21, 2024, 6:34am UTC](https://discuss.elastic.co/t/latency-in-seeing-a-document/363071/33 "2024-07-21T06:34:20Z")
```auto PUT /myindex-v2-2024.07.21/_create/g2210 { "timestamp": "2024-07-21T06:13:14.114Z", "log": { "timestamp": "2024-07-21T06:13:14.114Z", "sof": 334, "dd": { "as": "asdfjhg aslkf gaskf g", "l": 3939} } } ``` Did this but still can't see the doc in search. Also used @timestamp. Sorry where Data Views did come from? Of course this is not what the thread is about. Production logs all have timestamp. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 21, 2024, 7:47am UTC](https://discuss.elastic.co/t/latency-in-seeing-a-document/363071/34 "2024-07-21T07:47:27Z")
> [@siakc](#): > > Also used @timestamp. You've set fields called `timestamp`, not `@timestamp`. The difference is important. ---
**Author:** ![siakc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/siakc/32/135480_2.png) [@siakc](https://discuss.elastic.co/u/siakc)\ **Post date:** [July 21, 2024, 8:17am UTC](https://discuss.elastic.co/t/latency-in-seeing-a-document/363071/35 "2024-07-21T08:17:47Z")
I meant I have set @timestamp also. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 21, 2024, 8:23am UTC](https://discuss.elastic.co/t/latency-in-seeing-a-document/363071/36 "2024-07-21T08:23:44Z")
Rather than monitor by inserting dummy documents, which will pollute the index, I generally prefer adding an ingest timestamp through an ingest pipeline to each document. The ingest timestamp will reflect when Elasticsearch processed the document and be in UTC based on server time. With this I can run a query or create a visualisation showing how data comes in over time. It also allows me to calculate and visualize ingest delay by calculating the difference between the original timestamp and the ingest timestamp and also break it down by source, log type etc. ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [July 21, 2024, 11:30am UTC](https://discuss.elastic.co/t/latency-in-seeing-a-document/363071/37 "2024-07-21T11:30:56Z")
Sure, an ingest timestamp is a good idea generally, but its pretty clear to me that _here specifically_ we have a lack of understanding from @siakc around some basic concepts (sorry if that's harsh, but read the whole thread and tell me I'm wrong).. Like I said in an earlier post, we're now more at the "introduction to elasticsearch" sort of level, rather than troubleshooting the original delay/latency, which I think we agree is unlikely to be at the elasticsearch part of this overall "solution". My advice to @siakc would be dependent on the business value here - if this is a critical part of his company's operations/solution, he likely needs to start looking at supported solutions, so he can ask paid-for-engineering-teams for help going forward, and/or up-skilling the internal support team(s). ---
**Author:** ![siakc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/siakc/32/135480_2.png) [@siakc](https://discuss.elastic.co/u/siakc)\ **Post date:** [July 21, 2024, 12:16pm UTC](https://discuss.elastic.co/t/latency-in-seeing-a-document/363071/38 "2024-07-21T12:16:52Z")
I should have applied timezone shift in my query apparently. So now I can see that the inserted log is immediately searchable. I still don't know why there is a delay. I guess I have my answer to my original question. There is no delay in searchability. Though I did my test in a test environment. May also do it in production. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 21, 2024, 12:28pm UTC](https://discuss.elastic.co/t/latency-in-seeing-a-document/363071/39 "2024-07-21T12:28:15Z")
I do not think I have ever seen an issue where indexed documents take a long time to show up as searchable unless the refresh interval has been incorrectly set, so was expecting this simple test to show that the issue likely is client side. I have seen isues where users have tried to index documents individually at high throughput rates instead of in bulk and that the overhead associated with this has prevented Elasticsearch from keeping up. This as far as I remember often shows as failed indexing requests to the client so should be noticable. ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [July 21, 2024, 10:49pm UTC](https://discuss.elastic.co/t/latency-in-seeing-a-document/363071/40 "2024-07-21T22:49:43Z")
@stephenb wrote 8 days ago ":Show us the complete document." which would have helped, as it would have shown the timestamp (maybe @timestamp) field. ".. are you accounting for timezone?" I understand "no" is the answer. But @siakc > I should have applied timezone shift in my query apparently. ... I still don't know why there is a delay ... There is no delay in searchability Not really following that, there is and likely never was any delay. Kibana has a setting for which _timezone_ queries are related to, e.g. UTC, Europe/Berlin, whatever. So, e.g., "last 15 minutes" is translated into 2 timestamps, "15 minutes ago" and "now" correctly. Now elasticsearch is clever enough to convert things on the fly, on assumption it's supplied with a timestamp including a timezone for the timestamp field of each document, usually by using a standard format like say ISO8601. e.g 2024-07-22T00:39:36+02:00 2024-07-21T22:39:36Z which both represent the same actual time. But if any pipelines are ingesting with ONLY timestamps (i.e. without timezone, or with inconsistent timezones) then you may get problems e.g. the 2 timestamps above are identical, but 2024-07-22T00:39:36+02:00 (Europe/Berlin time) 2024-07-21T22:39:36Z (means UTC) 2024-07-21T22:39:36 These 3 are potentially not identical as the third one is ONLY the same actual time in Europe/Berlin timezones, anywhere else on the world that represents a different actual time. [Previous page](https://discuss.elastic.co/t/latency-in-seeing-a-document/363071.md?page=1) [Next page](https://discuss.elastic.co/t/latency-in-seeing-a-document/363071.md?page=3)