Not able to process large lines of log data into elasticsearch

[ggajanan]

Hello,

I have filebeat harvesting the log files and feeding it to logstash and then to the elasticsearch. There is one event which has 2185 lines into it. Configured, multiline.max_lines: 3000 , in filebeat.yml file. The logstash config is -

output {
elasticsearch {
action => "index"
hosts => "localhost:9200"
index => "logstash-dev"
flush_size => 2048
user => "logstash"
password => "password"
}
stdout {
codec => rubydebug
}
}

The logstash console shows the complete event getting processed but cannot find it in elasticsearch via Kibana console. All other smaller events are processed and can be seen successfully in kibana.

Please help me out on configuration I need to make the event appear in elasticsearch?

Thank you

[mainec (Isabel Drost-Fromm)]

Are there any errors in your ES log files?

[ggajanan]

Thank you. I found following error in the elasticsearch.log file.

java.lang.IllegalArgumentException: Document contains at least one immense term in field="logmessage.keyword" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '[35, 67, 65, 84, 67, 72, 35, 32, 99, 111, 109, 46, 115, 49, 46, 97, 112, 112, 46, 98, 97, 110, 107, 46, 97, 99, 99, 111, 117, 110]...', original message: bytes can be at most 32766 in length; got 35786
at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:772) ~[lucene-core-6.2.1.jar:6.2.1 43ab70147eb494324a1410f7a9f16a896a59bc6f - shalin - 2016-09-15 05:15:20]
at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:417) ~[lucene-core-6.2.1.jar:6.2.1 43ab70147eb494324a1410f7a9f16a896a59bc6f - shalin - 2016-09-15 05:15:20]
.
.
at org.elasticsearch.action.bulk.TransportShardBulkAction.onPrimaryShard(TransportShardBulkAction.java:74) [elasticsearch-5.0.1.jar:5.0.1]
.
.
Caused by: org.apache.lucene.util.BytesRefHash$MaxBytesLengthExceededException: bytes can be at most 32766 in length; got 35786
at org.apache.lucene.util.BytesRefHash.add(BytesRefHash.java:263) ~[lucene-core-6.2.1.jar:6.2.1 43ab70147eb494324a1410f7a9f16a896a59bc6f - shalin - 2016-09-15 05:15:20]

[mainec (Isabel Drost-Fromm)]

@ggajanan

Document contains at least one immense term

Searching for that error gave me the following two results that might help you further:

UTF8 encoding is longer than the max length 32766

You can try ignore_above: 256 From the docs; ignore_above The analyzer will ignore strings larger than this size. Useful for generic not_analyzed fields that should ignore long text.

UTF8 encoding is longer than the max length 32766

Hope this helps,
Isabel

[ggajanan]

Thank you so much for all your help.