# Not able to process large lines of log data into elasticsearch

**URL:** <https://discuss.elastic.co/t/not-able-to-process-large-lines-of-log-data-into-elasticsearch/66928>\
**Category:** Elasticsearch\
**Created:** [November 23, 2016, 2:41am UTC](https://discuss.elastic.co/t/not-able-to-process-large-lines-of-log-data-into-elasticsearch/66928 "2016-11-23T02:41:22Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![ggajanan](https://avatars.discourse-cdn.com/v4/letter/g/838e76/32.png) [@ggajanan](https://discuss.elastic.co/u/ggajanan)\
**Post date:** [November 23, 2016, 2:41am UTC](https://discuss.elastic.co/t/not-able-to-process-large-lines-of-log-data-into-elasticsearch/66928/1 "2016-11-23T02:41:22Z")

</div>

Hello,

I have filebeat harvesting the log files and feeding it to logstash and then to the elasticsearch. There is one event which has 2185 lines into it. Configured, multiline.max\_lines: 3000 , in filebeat.yml file. The logstash config is -

output {  
elasticsearch {  
action =\> "index"  
hosts =\> "localhost:9200"  
index =\> "logstash-dev"  
flush\_size =\> 2048  
user =\> "logstash"  
password =\> "password"  
}  
stdout {  
codec =\> rubydebug  
}  
}

The logstash console shows the complete event getting processed but cannot find it in elasticsearch via Kibana console. All other smaller events are processed and can be seen successfully in kibana.

Please help me out on configuration I need to make the event appear in elasticsearch?

Thank you

---

<div class="post-metadata">

**Author:** ![mainec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mainec/32/5557_2.png) [@mainec](https://discuss.elastic.co/u/mainec)\
**Post date:** [November 23, 2016, 11:14am UTC](https://discuss.elastic.co/t/not-able-to-process-large-lines-of-log-data-into-elasticsearch/66928/2 "2016-11-23T11:14:57Z")

</div>

Are there any errors in your ES log files?

---

<div class="post-metadata">

**Author:** ![ggajanan](https://avatars.discourse-cdn.com/v4/letter/g/838e76/32.png) [@ggajanan](https://discuss.elastic.co/u/ggajanan)\
**Post date:** [November 23, 2016, 12:48pm UTC](https://discuss.elastic.co/t/not-able-to-process-large-lines-of-log-data-into-elasticsearch/66928/3 "2016-11-23T12:48:35Z")

</div>

Thank you. I found following error in the elasticsearch.log file.

java.lang.IllegalArgumentException: Document contains at least one immense term in field="logmessage.keyword" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '[35, 67, 65, 84, 67, 72, 35, 32, 99, 111, 109, 46, 115, 49, 46, 97, 112, 112, 46, 98, 97, 110, 107, 46, 97, 99, 99, 111, 117, 110]...', original message: bytes can be at most 32766 in length; got 35786  
at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:772) ~[lucene-core-6.2.1.jar:6.2.1 43ab70147eb494324a1410f7a9f16a896a59bc6f - shalin - 2016-09-15 05:15:20]  
at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:417) ~[lucene-core-6.2.1.jar:6.2.1 43ab70147eb494324a1410f7a9f16a896a59bc6f - shalin - 2016-09-15 05:15:20]  
.  
.  
at org.elasticsearch.action.bulk.TransportShardBulkAction.onPrimaryShard(TransportShardBulkAction.java:74) [elasticsearch-5.0.1.jar:5.0.1]  
.  
.  
Caused by: org.apache.lucene.util.BytesRefHash$MaxBytesLengthExceededException: bytes can be at most 32766 in length; got 35786  
at org.apache.lucene.util.BytesRefHash.add(BytesRefHash.java:263) ~[lucene-core-6.2.1.jar:6.2.1 43ab70147eb494324a1410f7a9f16a896a59bc6f - shalin - 2016-09-15 05:15:20]

---

<div class="post-metadata">

**Author:** ![mainec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mainec/32/5557_2.png) [@mainec](https://discuss.elastic.co/u/mainec)\
**Post date:** [November 23, 2016, 1:26pm UTC](https://discuss.elastic.co/t/not-able-to-process-large-lines-of-log-data-into-elasticsearch/66928/4 "2016-11-23T13:26:53Z")

</div>

> [@ggajanan](#):
>
> Document contains at least one immense term

Searching for that error gave me the following two results that might help you further:

> [@UTF8 encoding is longer than the max length 32766](https://discuss.elastic.co/t/utf8-encoding-is-longer-than-the-max-length-32766/816/2):
>
> You can try ignore\_above: 256 From [the docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-core-types.html#mapping-core-types); ignore\_above The analyzer will ignore strings larger than this size. Useful for generic not\_analyzed fields that should ignore long text.

> <https://stackoverflow.com/questions/24019868/utf8-encoding-is-longer-than-the-max-length-32766>

Hope this helps,  
Isabel

---

<div class="post-metadata">

**Author:** ![ggajanan](https://avatars.discourse-cdn.com/v4/letter/g/838e76/32.png) [@ggajanan](https://discuss.elastic.co/u/ggajanan)\
**Post date:** [November 23, 2016, 5:09pm UTC](https://discuss.elastic.co/t/not-able-to-process-large-lines-of-log-data-into-elasticsearch/66928/5 "2016-11-23T17:09:09Z")

</div>

Thank you so much for all your help. 👍

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 21, 2016, 5:09pm UTC](https://discuss.elastic.co/t/not-able-to-process-large-lines-of-log-data-into-elasticsearch/66928/6 "2016-12-21T17:09:23Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.