Unable to send logs to elasticsearch due to max_bytes_length_exceeded_exception

Observing below log in Fluentd with max_bytes_length_exceeded_exception. Please help to identify the root cause and fix

2022-06-22 22:18:30 +0800 [error]: #0 Could not push log to Elasticsearch: {"took"=>30, "errors"=>true, "items"=>[{"index"=>{"_index"=>"org-application-dms-2022.06.22", "_type"=>"_doc", "_id"=>"hTzHi4EBXbl5SkL7aLFN", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Document contains at least one immense term in field=\"message.keyword\" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped.  Please correct the analyzer to not produce such terms.  The prefix of the first immense term is: '[50, 48, 50, 50, 45, 48, 54, 45, 50, 50, 32, 50, 48, 58, 51, 57, 58, 51, 49, 46, 55, 53, 49, 124, 91, 121, 112, 101, 67, 111]...', original message: bytes can be at most 32766 in length; got 217760", "caused_by"=>{"type"=>"max_bytes_length_exceeded_exception", "reason"=>"max_bytes_length_exceeded_exception: bytes can be at most 32766 in length; got 217760"}}}}, {"index"=>{"_index"=>"org-application-dms-2022.06.22", "_type"=>"_doc", "_id"=>"hjzHi4EBXbl5SkL7aLFN", "_version"=>1, "result"=>"created", "_shards"=>{"total"=>2, "successful"=>2, "failed"=>0}, "_seq_no"=>431702, "_primary_term"=>1, "status"=>201}}, {"index"=>{"_index"=>"org-application-dms-2022.06.22", "_type"=>"_doc", "_id"=>"hzzHi4EBXbl5SkL7aLFN", "_version"=>1, "result"=>"created", "_shards"=>{"total"=>2, "successful"=>2, "failed"=>0}, "_seq_no"=>431703, "_primary_term"=>1, "status"=>201}}]}

Hi @Gihan_Indrajith

According to the documentation:

This option is also useful for protecting against Lucene’s term byte-length limit of 32766 .

Try use ignore_above.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.