# Handling persistent ES output failures **URL:** \ **Category:** Logstash\ **Created:** [February 24, 2017, 11:10am UTC](https://discuss.elastic.co/t/handling-persistent-es-output-failures/76387 "2017-02-24T11:10:08Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Dan\_Markhasin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dan_markhasin/32/14187_2.png) [@Dan\_Markhasin](https://discuss.elastic.co/u/Dan_Markhasin)\ **Post date:** [February 24, 2017, 11:10am UTC](https://discuss.elastic.co/t/handling-persistent-es-output-failures/76387/1 "2017-02-24T11:10:08Z")
Hi, I am trying to understand if there is a method in logstash to handle failures that occur when writing to ElasticSearch. I know that logstash has a retry policy for certain types of exceptions, but the failures I am looking to handle are permanent failures, such as ES rejecting an event (because maybe the index field has an uppercase character, or maybe the target index doesn't exist in ES, etc.) In such cases the document will never be accepted by ES. Is there any way to monitor for this condition, or potentially use a different output for events that were rejected by ES? ---
**Author:** ![lueneburger](https://avatars.discourse-cdn.com/v4/letter/l/f475e1/32.png) [@lueneburger](https://discuss.elastic.co/u/lueneburger)\ **Post date:** [February 24, 2017, 11:28am UTC](https://discuss.elastic.co/t/handling-persistent-es-output-failures/76387/2 "2017-02-24T11:28:40Z")
Hi Dan did you try to add a tag on failure? for example: ``` filter { grok { match => ["message", "your filter"] tag_on_failure => ["failed_to_filter"] } } output { if "failed_to_filter" in [tags] { file { path => ... codec => line { format => "custom format: %{message}"} } } } ``` don't know if there is an even better way, that's how I would try it [tag\_on\_failure](https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html#plugins-filters-grok-tag_on_failure) information ---
**Author:** ![Dan\_Markhasin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dan_markhasin/32/14187_2.png) [@Dan\_Markhasin](https://discuss.elastic.co/u/Dan_Markhasin)\ **Post date:** [February 24, 2017, 11:42am UTC](https://discuss.elastic.co/t/handling-persistent-es-output-failures/76387/3 "2017-02-24T11:42:12Z")
This would help to identify events that failed to parse by Grok, not events that are being rejected by ES itself. Consider the following output configuration: ```auto output { elasticsearch { hosts => ["http://localhost:9200"] document_type => "doc" document_id => "%{doc_id}" index => "%{index}-%{+YYYY.MM.dd}" } } ``` Now the following event arrives: { "doc\_id":12345. "index":"MYINDEX", "value":"some\_string" } This is a perfectly valid event from logstash's perspective, it parses just fine in grok, but ES is going to reject it because the index is in uppercase... and there doesn't seem to be any way to detect this type of failure. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [February 28, 2017, 7:03am UTC](https://discuss.elastic.co/t/handling-persistent-es-output-failures/76387/4 "2017-02-28T07:03:52Z")
Yeah, I don't think this there's much to do about this at the moment. I think a dead letter feature is in the works. ---
**Author:** ![Dan\_Markhasin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dan_markhasin/32/14187_2.png) [@Dan\_Markhasin](https://discuss.elastic.co/u/Dan_Markhasin)\ **Post date:** [February 28, 2017, 7:51pm UTC](https://discuss.elastic.co/t/handling-persistent-es-output-failures/76387/5 "2017-02-28T19:51:30Z")
Is there some way to print the JSON of such failed events to the log? That way at least we'll be able to pick them up by combing the logs... ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [March 1, 2017, 6:53am UTC](https://discuss.elastic.co/t/handling-persistent-es-output-failures/76387/6 "2017-03-01T06:53:19Z")
Failures will be noted in the log, but it'll be in a Ruby representation that's less fun to parse. ---
**Author:** ![Dan\_Markhasin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dan_markhasin/32/14187_2.png) [@Dan\_Markhasin](https://discuss.elastic.co/u/Dan_Markhasin)\ **Post date:** [March 1, 2017, 7:11am UTC](https://discuss.elastic.co/t/handling-persistent-es-output-failures/76387/7 "2017-03-01T07:11:28Z")
From what I've seen the failures look like this (for example): [2017-02-28T18:54:27,878][WARN][logstash.outputs.elasticsearch] Failed action. {:status=\>400, :action=\>["index", {:\_id=\>nil, :\_index=\>"KAFKA-LAG-2017.02.28", :\_type=\>"lag", :\_routing=\>nil}, 2017-02-28T18:54:27.797Z 10.11.103.26 %{message}], :response=\>{"index"=\>{"\_index"=\>"KAFKA-LAG-2017.02.28", "\_type"=\>"lag", "\_id"=\>nil, "status"=\>400, "error"=\>{"type"=\>"invalid\_index\_name\_exception", "reason"=\>"Invalid index name [KAFKA-LAG-2017.02.28], must be lowercase", "index\_uuid"=\>"_na_", "index"=\>"KAFKA-LAG-2017.02.28"}}}} There is really no information in this log entry that would allow me to identify the failed record... ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [March 1, 2017, 7:12am UTC](https://discuss.elastic.co/t/handling-persistent-es-output-failures/76387/8 "2017-03-01T07:12:27Z")
Oh, my bad. I must've conflated that error message with other cases when the document _is_ logged. This is arguably a bug. ---
**Author:** ![Dan\_Markhasin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dan_markhasin/32/14187_2.png) [@Dan\_Markhasin](https://discuss.elastic.co/u/Dan_Markhasin)\ **Post date:** [March 1, 2017, 10:16am UTC](https://discuss.elastic.co/t/handling-persistent-es-output-failures/76387/9 "2017-03-01T10:16:24Z")
Ok, I'll submit this to the GitHub project then 🙂 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 29, 2017, 10:16am UTC](https://discuss.elastic.co/t/handling-persistent-es-output-failures/76387/10 "2017-03-29T10:16:35Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.