# Logstash to Elasticsearch there is 10-20min for delay, also not all logs are indexed

**URL:** <https://discuss.elastic.co/t/logstash-to-elasticsearch-there-is-10-20min-for-delay-also-not-all-logs-are-indexed/336241>\
**Category:** Logstash\
**Created:** [June 16, 2023, 2:54pm UTC](https://discuss.elastic.co/t/logstash-to-elasticsearch-there-is-10-20min-for-delay-also-not-all-logs-are-indexed/336241 "2023-06-16T14:54:26Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![mayank\_singh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mayank_singh/32/122370_2.png) [@mayank\_singh](https://discuss.elastic.co/u/mayank_singh)\
**Post date:** [June 16, 2023, 2:54pm UTC](https://discuss.elastic.co/t/logstash-to-elasticsearch-there-is-10-20min-for-delay-also-not-all-logs-are-indexed/336241/1 "2023-06-16T14:54:26Z")

</div>

Hi,  
I am sending logs from Logstash to Elasticsearch. The Elasticsearch seems to be working fine but there is 10-20mins of delay in logs index also getting below error on logstash, any help would be greatly appreciated.

```auto
[2023-06-16T14:36:50,078][WARN][logstash.outputs.elasticsearch][main][044debbbdfef92ace7e72d6a3ec877042a9516199b68b4ae9f8ddd3c0a5bc642] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-2023.06.16", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x42680064>], :response=>{"index"=>{"_index"=>"logstash-2023.06.16", "_type"=>"_doc", "_id"=>"VKujxIgBjcUFivZxMzvM", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [timestamp] of type [date] in document with id 'VKujxIgBjcUFivZxMzvM'. Preview of field's value: '1686925083.1633325'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"failed to parse date field [1686925083.1633325] with format [strict_date_optional_time||epoch_millis]", "caused_by"=>{"type"=>"date_time_parse_exception", "reason"=>"date_time_parse_exception: Failed to parse with all enclosed parsers"}}}}}}

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 16, 2023, 4:43pm UTC](https://discuss.elastic.co/t/logstash-to-elasticsearch-there-is-10-20min-for-delay-also-not-all-logs-are-indexed/336241/2 "2023-06-16T16:43:58Z")

</div>

> [@mayank\_singh](#):
>
> `1686925083.1633325`

I cannot speak to the delay without seeing more of the configuration, but I think the failure to parse '1686925083.1633325' as [strict\_date\_optional\_time||epoch\_millis] is because that time is in epoch\_second, not epoch\_millis.

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [June 17, 2023, 10:49am UTC](https://discuss.elastic.co/t/logstash-to-elasticsearch-there-is-10-20min-for-delay-also-not-all-logs-are-indexed/336241/3 "2023-06-17T10:49:22Z")

</div>

Just to add, I assume issues are: grok(especially if there is too many GREEDYDATA) and mapper\_parsing\_exception.

---

<div class="post-metadata">

**Author:** ![Hemanth\_Gowda](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hemanth_gowda/32/23886_2.png) [@Hemanth\_Gowda](https://discuss.elastic.co/u/Hemanth_Gowda)\
**Post date:** [June 17, 2023, 3:20pm UTC](https://discuss.elastic.co/t/logstash-to-elasticsearch-there-is-10-20min-for-delay-also-not-all-logs-are-indexed/336241/4 "2023-06-17T15:20:49Z")

</div>

Please check if you have refresh interval set to your index which is causing the delay.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 15, 2023, 3:21pm UTC](https://discuss.elastic.co/t/logstash-to-elasticsearch-there-is-10-20min-for-delay-also-not-all-logs-are-indexed/336241/5 "2023-07-15T15:21:37Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.