Hello Community,
in a logstash / elasticsearch pipeline I notice rare errors similar to this one:
[2020-06-08T17:02:05,621][WARN ][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-audit-2020.06.08", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x68fc4f79>], :response=>{"index"=>{"_index"=>"logstash-audit-2020.06.08", "_type"=>"_doc", "_id"=>"P_A9lHIB1S6G3GE4Ad6k", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Document contains at least one immense term in field=\"message\" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '[60, 63, 120, 109, 108, 32, 118, 101, 114, 115, 105, 111, 110, 61, 34, 49, 46, 48, 34, 32, 101, 110, 99, 111, 100, 105, 110, 103, 61, 34]...', original message: bytes can be at most 32766 in length; got 72253", "caused_by"=>{"type"=>"max_bytes_length_exceeded_exception", "reason"=>"max_bytes_length_exceeded_exception: bytes can be at most 32766 in length; got 72253"}}}}}
But I struggle to find what was the exact message that caused this behaviour. Can you help me:
-
is it possible to configure logstash to output such "non-indexed" messages to file/log so that they can be analysed later - i.e. by some config option or by some logging configuration setting?
-
of course if you have an idea with respect to solving the error itself by i.e. modifying the logstash pipeline - then this is also super interesting .. by i.e. increasing the 32k limit or avoiding it..
the related logstash.yml pipeline is here: https://github.com/dcm4che-dockerfiles/logstash-dcm4chee/blob/master/logstash.yml
thanks a lot!!