Not able to process large lines of log data into elasticsearch

Elastic Stack Elasticsearch
1

Hello,

I have filebeat harvesting the log files and feeding it to logstash and then to the elasticsearch. There is one event which has 2185 lines into it. Configured, multiline.max_lines: 3000 , in filebeat.yml file. The logstash config is -

output {
elasticsearch {
action => "index"
hosts => "localhost:9200"
index => "logstash-dev"
flush_size => 2048
user => "logstash"
password => "password"
}
stdout {
codec => rubydebug
}
}

The logstash console shows the complete event getting processed but cannot find it in elasticsearch via Kibana console. All other smaller events are processed and can be seen successfully in kibana.

Please help me out on configuration I need to make the event appear in elasticsearch?

Thank you

2

Are there any errors in your ES log files?

3

Thank you. I found following error in the elasticsearch.log file.

java.lang.IllegalArgumentException: Document contains at least one immense term in field="logmessage.keyword" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '[35, 67, 65, 84, 67, 72, 35, 32, 99, 111, 109, 46, 115, 49, 46, 97, 112, 112, 46, 98, 97, 110, 107, 46, 97, 99, 99, 111, 117, 110]...', original message: bytes can be at most 32766 in length; got 35786
at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:772) ~[lucene-core-6.2.1.jar:6.2.1 43ab70147eb494324a1410f7a9f16a896a59bc6f - shalin - 2016-09-15 05:15:20]
at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:417) ~[lucene-core-6.2.1.jar:6.2.1 43ab70147eb494324a1410f7a9f16a896a59bc6f - shalin - 2016-09-15 05:15:20]
.
.
at org.elasticsearch.action.bulk.TransportShardBulkAction.onPrimaryShard(TransportShardBulkAction.java:74) [elasticsearch-5.0.1.jar:5.0.1]
.
.
Caused by: org.apache.lucene.util.BytesRefHash$MaxBytesLengthExceededException: bytes can be at most 32766 in length; got 35786
at org.apache.lucene.util.BytesRefHash.add(BytesRefHash.java:263) ~[lucene-core-6.2.1.jar:6.2.1 43ab70147eb494324a1410f7a9f16a896a59bc6f - shalin - 2016-09-15 05:15:20]

4

Searching for that error gave me the following two results that might help you further:

Hope this helps,
Isabel

1 Like
5

Thank you so much for all your help. :thumbsup:

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.