# Almost 4 hour delay in documents being received and showing up in Kibana Discover **URL:** \ **Category:** Elasticsearch\ **Created:** [April 2, 2019, 7:59pm UTC](https://discuss.elastic.co/t/almost-4-hour-delay-in-documents-being-received-and-showing-up-in-kibana-discover/175067 "2019-04-02T19:59:18Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![xq1xq1xq1](https://avatars.discourse-cdn.com/v4/letter/x/82dd89/32.png) [@xq1xq1xq1](https://discuss.elastic.co/u/xq1xq1xq1)\ **Post date:** [April 2, 2019, 7:59pm UTC](https://discuss.elastic.co/t/almost-4-hour-delay-in-documents-being-received-and-showing-up-in-kibana-discover/175067/1 "2019-04-02T19:59:19Z")
Kibana: Version: 5.6.13 ElasticSearch: 5.6.13 Ubuntu 18.04 CPUs: 4 Memory: 28 GB This is running inside a QEMU KVM VM and the box is not heavily taxed. ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/d/5de01cf71eca218e6ea5eb247ffabac0254f3a21.png) I ingesting live packets into elasticsearch using tshark. When I query using Kibana the last record is Current Kibana Latest Record: April 2nd 2019, 11:00:42.151 Current Time: Apr 2 13:56:45 CST 2019 This is a delay of about 2 hours and 56 minutes Ideas on how to help speed up the indexing or am I missing something else? ---
**Author:** ![xq1xq1xq1](https://avatars.discourse-cdn.com/v4/letter/x/82dd89/32.png) [@xq1xq1xq1](https://discuss.elastic.co/u/xq1xq1xq1)\ **Post date:** [April 5, 2019, 5:06pm UTC](https://discuss.elastic.co/t/almost-4-hour-delay-in-documents-being-received-and-showing-up-in-kibana-discover/175067/2 "2019-04-05T17:06:50Z")
Nobody has any suggestions?? please! ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [April 5, 2019, 9:16pm UTC](https://discuss.elastic.co/t/almost-4-hour-delay-in-documents-being-received-and-showing-up-in-kibana-discover/175067/3 "2019-04-05T21:16:59Z")
What does an example document look like? What timezone are you in? How is the timezone reflected in the document's timestamp? I'm wondering if the ingested data has a timestamp in the "future" ---
**Author:** ![xq1xq1xq1](https://avatars.discourse-cdn.com/v4/letter/x/82dd89/32.png) [@xq1xq1xq1](https://discuss.elastic.co/u/xq1xq1xq1)\ **Post date:** [April 9, 2019, 7:47pm UTC](https://discuss.elastic.co/t/almost-4-hour-delay-in-documents-being-received-and-showing-up-in-kibana-discover/175067/4 "2019-04-09T19:47:34Z")
The documents are actually packets being sent from tshark. my timezone is GMT-6 This is an example packet: {"index" : {"\_index": "packets-2019-04-09", "\_type": "pcap\_file", "\_score": null}} {"timestamp" : "1554838586894", "layers" : {"frame\_raw": "0", "frame": {"filtered": "frame"},"eth\_raw": "00","eth": {"filtered": "eth"},"ip\_raw": "4","ip": {"filtered": "ip"}, "sctp\_raw": "5","sctp": {"filtered": "sctp"},"m2pa\_raw": "00", "m2pa": {"filtered": "m2pa"},"mtp3\_raw": "a","mtp3": {"filtered": "mtp3"}, "sccp\_raw": "s", "sccp": {"filtered": "sccp"}}} The issue is that the last packet available in Kibana keeps on falling behind the current time even though I see packets being received by ElasticSearch. Latest document available on Kibana Current date on server March 27th 2019, 04:15:15.942 Wed Mar 27 09:49:19 CST 2019 5:34 behind March 28th 2019, 05:17:43.211 Thu Mar 28 12:07:49 CST 2019 6:50 behind March 28th 2019, 08:06:19.134 Thu Mar 28 15:38:19 CST 2019 7:32 behind But at the same time I am seeing the number of documents being added to the index increasing: Documents 1,294,179 1,294,179 Documents 1,296,465 1,296,465 Thanks so much for reaching out. Ideas? ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [April 9, 2019, 9:08pm UTC](https://discuss.elastic.co/t/almost-4-hour-delay-in-documents-being-received-and-showing-up-in-kibana-discover/175067/5 "2019-04-09T21:08:04Z")
Maybe check to see if your data is getting into Elasticsearch. For example, see what docs that have a timestamp in the last 5 minutes: ```auto GET packets-2019-04-09/_search { "query": { "bool": { "filter": [ { "range" : { "timestamp" : { "gte": "now-5m" } } } ] } } } ``` ---
**Author:** ![xq1xq1xq1](https://avatars.discourse-cdn.com/v4/letter/x/82dd89/32.png) [@xq1xq1xq1](https://discuss.elastic.co/u/xq1xq1xq1)\ **Post date:** [April 10, 2019, 11:49am UTC](https://discuss.elastic.co/t/almost-4-hour-delay-in-documents-being-received-and-showing-up-in-kibana-discover/175067/6 "2019-04-10T11:49:29Z")
I tried this: > curl -XPOST 'localhost:9200/GET packets-2019-04-09/\_search > > > { > > "query": { > > "bool": { > > "filter": [ > > { "range" : { "timestamp" : { "gte": "now-5m" } } } > > > > ``` > > ] > > } > > } > > > > ``` > > > > }' But got this error: > curl: (3) [globbing] nested brace in column 62 Ideas as to what I did wrong? Thanks so much for your insights! ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [April 10, 2019, 12:20pm UTC](https://discuss.elastic.co/t/almost-4-hour-delay-in-documents-being-received-and-showing-up-in-kibana-discover/175067/7 "2019-04-10T12:20:34Z")
No, you enter the text exactly as shown in the DevTools console: ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/7/f74cfcefdedcd2c118c242a4b92b8f3680a3f365.png) If you want to use `curl` you need to format the request differently. Probably easier for you to use DevTools console ---
**Author:** ![xq1xq1xq1](https://avatars.discourse-cdn.com/v4/letter/x/82dd89/32.png) [@xq1xq1xq1](https://discuss.elastic.co/u/xq1xq1xq1)\ **Post date:** [April 10, 2019, 2:32pm UTC](https://discuss.elastic.co/t/almost-4-hour-delay-in-documents-being-received-and-showing-up-in-kibana-discover/175067/8 "2019-04-10T14:32:19Z")
Thanks so much for your assistance - I am very new to this and this is extremely helpful advice. I do not see any hits until I go back to now-8h using the file packets-2019-04-10. But I do see hits consistently prior to 8h. To me this seems to indicate that the indexing of the incoming data is occurring is but is requiring more than resources than available in real-time to ingest and thus the processing of the data is getting backed up in the system. Is this a correct understanding? This is the current load on the host: ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/3/13f8d600499b20f91a0595b1fcb2c193475421ae.png) The host does not seem to be overly taxed. This is the current elasticsearch status: ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/f/cf96f486a90bc0ea02c5b4d03a7affdc7a1b531e.png) Any suggestions on how to resolve this lag in getting the documents into kibana? Thanks again for imparting your wisdom in trying to resolve this issue. ---
**Author:** ![xq1xq1xq1](https://avatars.discourse-cdn.com/v4/letter/x/82dd89/32.png) [@xq1xq1xq1](https://discuss.elastic.co/u/xq1xq1xq1)\ **Post date:** [April 10, 2019, 2:52pm UTC](https://discuss.elastic.co/t/almost-4-hour-delay-in-documents-being-received-and-showing-up-in-kibana-discover/175067/9 "2019-04-10T14:52:46Z")
I just determined that the time stamps in the data is in epoch. This means that the delay is 2 hours not 8 hours so not quite as bad but would still like to make it near real time if possible. ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [April 10, 2019, 3:35pm UTC](https://discuss.elastic.co/t/almost-4-hour-delay-in-documents-being-received-and-showing-up-in-kibana-discover/175067/10 "2019-04-10T15:35:47Z")
Right, I certainly had a feeling that timezones had at least some type of influence on this situation. Again, however, it is still unlikely that Elasticsearch is queuing up data for 2 hours. What is the mechanism you are using to connect tshark to Elasticsearch. A few approaches are described in this blog: [https://www.elastic.co/blog/analyzing-network-packets-with-wireshark-elasticsearch-and-kibana](https://www.elastic.co/blog/analyzing-network-packets-with-wireshark-elasticsearch-and-kibana) What are you doing? Curl? Filebeat? Logstash? ---
**Author:** ![xq1xq1xq1](https://avatars.discourse-cdn.com/v4/letter/x/82dd89/32.png) [@xq1xq1xq1](https://discuss.elastic.co/u/xq1xq1xq1)\ **Post date:** [April 10, 2019, 6:43pm UTC](https://discuss.elastic.co/t/almost-4-hour-delay-in-documents-being-received-and-showing-up-in-kibana-discover/175067/11 "2019-04-10T18:43:25Z")
I am piping tshark to curl: > tshark -i ens5 -T ek -l | curl.sh Any thoughts on how to investigate this further? ---
**Author:** ![xq1xq1xq1](https://avatars.discourse-cdn.com/v4/letter/x/82dd89/32.png) [@xq1xq1xq1](https://discuss.elastic.co/u/xq1xq1xq1)\ **Post date:** [April 10, 2019, 6:45pm UTC](https://discuss.elastic.co/t/almost-4-hour-delay-in-documents-being-received-and-showing-up-in-kibana-discover/175067/12 "2019-04-10T18:45:58Z")
This is my elasticsearch summary ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/5/756c1295e214c8d559fe48cf2a887eb8b0e781c5.png) Any concerns with this information for the performance of elasticsearch? ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [April 11, 2019, 2:34pm UTC](https://discuss.elastic.co/t/almost-4-hour-delay-in-documents-being-received-and-showing-up-in-kibana-discover/175067/13 "2019-04-11T14:34:26Z")
> [@xq1xq1xq1](#): > > I am piping tshark to curl: What's inside `curl.sh`? ---
**Author:** ![xq1xq1xq1](https://avatars.discourse-cdn.com/v4/letter/x/82dd89/32.png) [@xq1xq1xq1](https://discuss.elastic.co/u/xq1xq1xq1)\ **Post date:** [April 11, 2019, 3:49pm UTC](https://discuss.elastic.co/t/almost-4-hour-delay-in-documents-being-received-and-showing-up-in-kibana-discover/175067/14 "2019-04-11T15:49:25Z")
Here you go: ```auto #!/bin/bash - i=0 while read line; do # process only non empty lines because of older tshark release if [! -z "$line"]; then c=$(printf '%s\n%s\n' "$c" "$line") i=$((i+1)) # curl only every X seconds if !((i % 2)) && !((SECONDS % 10)) && [[-v c]]; then #printf '%s\n' "$c" printf '%s\n' "$c" | curl -o /dev/null --silent -XPUT http://localhost:9200/_bulk --data-binary @- & c= i=0 fi fi done #echo $c if [[-v c]]; then printf '%s\n' "$c" | curl -o /dev/null --silent -XPUT http://localhost:9200/_bulk --data-binary @- & fi ``` ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [April 11, 2019, 4:53pm UTC](https://discuss.elastic.co/t/almost-4-hour-delay-in-documents-being-received-and-showing-up-in-kibana-discover/175067/15 "2019-04-11T16:53:08Z")
Maybe allow the shell script to also echo lines to stdout and check the timestamp on some of the records. Inspect the timestamp (decode from epoch time) and see if the value makes sense (i.e. does it match up to "now"?) Then, as soon as possible, query Elasticsearch for records matching that timestamp exactly. Something like: ```auto GET packets-2019-04-11/_search { "query": { "bool": { "filter": [ { "term" : { "timestamp" : "1554044059" } } ] } } } ``` (or the ISO-8601 version of the timestamp). Also, do you possibly have 2 timestamps in your index? That is, `timestamp` and `@timestamp`? ---
**Author:** ![xq1xq1xq1](https://avatars.discourse-cdn.com/v4/letter/x/82dd89/32.png) [@xq1xq1xq1](https://discuss.elastic.co/u/xq1xq1xq1)\ **Post date:** [April 15, 2019, 4:26pm UTC](https://discuss.elastic.co/t/almost-4-hour-delay-in-documents-being-received-and-showing-up-in-kibana-discover/175067/16 "2019-04-15T16:26:34Z")
You are da man! I am sending the curl output into a log file then I am tailing the file. As soon as I get a record on the screen, I then query that timestamp in Kibana and the record is there. The time between the record in kibana is still drifting away from the current time though. ---
**Author:** ![xq1xq1xq1](https://avatars.discourse-cdn.com/v4/letter/x/82dd89/32.png) [@xq1xq1xq1](https://discuss.elastic.co/u/xq1xq1xq1)\ **Post date:** [April 24, 2019, 7:17pm UTC](https://discuss.elastic.co/t/almost-4-hour-delay-in-documents-being-received-and-showing-up-in-kibana-discover/175067/17 "2019-04-24T19:17:54Z")
Just restarted the service and the time stamps seem to coincide within a few seconds from the server to the packet time being ingested at the start. This time immediately started drifting apart to reach a few minutes in very short order with the difference still increasing. Any thoughts as to how to ingest the data more in real-time? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 22, 2019, 7:18pm UTC](https://discuss.elastic.co/t/almost-4-hour-delay-in-documents-being-received-and-showing-up-in-kibana-discover/175067/18 "2019-05-22T19:18:00Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.