Handling persistent ES output failures

[Dan_Markhasin]

Hi,

I am trying to understand if there is a method in logstash to handle failures that occur when writing to ElasticSearch.
I know that logstash has a retry policy for certain types of exceptions, but the failures I am looking to handle are permanent failures, such as ES rejecting an event (because maybe the index field has an uppercase character, or maybe the target index doesn't exist in ES, etc.)
In such cases the document will never be accepted by ES.

Is there any way to monitor for this condition, or potentially use a different output for events that were rejected by ES?

[lueneburger (Dirk L.)]

Hi Dan

did you try to add a tag on failure?

for example:

filter {

	grok {
			match          => ["message", "your filter"]
			tag_on_failure => [ "failed_to_filter" ]
		}
}


output {

	if "failed_to_filter" in [tags] {

		file {
		path => ...
		codec => line { format => "custom format: %{message}"}
		}
	}
}

don't know if there is an even better way, that's how I would try it

tag_on_failure information

[Dan_Markhasin]

This would help to identify events that failed to parse by Grok, not events that are being rejected by ES itself.
Consider the following output configuration:

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    document_type => "doc"
    document_id => "%{doc_id}"
    index => "%{index}-%{+YYYY.MM.dd}"
  }
}

Now the following event arrives:
{
"doc_id":12345.
"index":"MYINDEX",
"value":"some_string"
}

This is a perfectly valid event from logstash's perspective, it parses just fine in grok, but ES is going to reject it because the index is in uppercase... and there doesn't seem to be any way to detect this type of failure.

[magnusbaeck (Magnus Bäck)]

Yeah, I don't think this there's much to do about this at the moment. I think a dead letter feature is in the works.

[Dan_Markhasin]

Is there some way to print the JSON of such failed events to the log? That way at least we'll be able to pick them up by combing the logs...

[magnusbaeck (Magnus Bäck)]

Failures will be noted in the log, but it'll be in a Ruby representation that's less fun to parse.

[Dan_Markhasin]

From what I've seen the failures look like this (for example):

[2017-02-28T18:54:27,878][WARN ][logstash.outputs.elasticsearch] Failed action. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"KAFKA-LAG-2017.02.28", :_type=>"lag", :_routing=>nil}, 2017-02-28T18:54:27.797Z 10.11.103.26 %{message}], :response=>{"index"=>{"_index"=>"KAFKA-LAG-2017.02.28", "_type"=>"lag", "_id"=>nil, "status"=>400, "error"=>{"type"=>"invalid_index_name_exception", "reason"=>"Invalid index name [KAFKA-LAG-2017.02.28], must be lowercase", "index_uuid"=>"na", "index"=>"KAFKA-LAG-2017.02.28"}}}}

There is really no information in this log entry that would allow me to identify the failed record...

[magnusbaeck (Magnus Bäck)]

Oh, my bad. I must've conflated that error message with other cases when the document is logged. This is arguably a bug.

[Dan_Markhasin]

Ok, I'll submit this to the GitHub project then