How to increase byte size in processing in logstash?

[Shweta]

Hi,

While filtering logs in logstash we came across an error which states reason as:

"reason"=>"max_bytes_length_exceeded_exception: bytes can be at most 32766 in length; got 56665"}}}}

We cannot tokenize our message field. Please let us know if there is any way to get rid of this limit, so that it will be able to filter mesages greater than 32kb.

Thanks,
Shweta

[jsvd (João Duarte)]

which filter is causing this? can you post your config? and the whole error log line?

[Shweta]

I am getting a big long error consisting of complete parsed log event with all details and below is the part which puts more light on the problem. Hope this helps you

"reason"=>"Document contains at least one immense term in field="message" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '[10, 10, 42, 42, 42, 42, 42, 42, 32, 99, 104, 97, 105, 110, 32, 42, 42, 42, 42, 42, 42, 10, 78, 97, 109, 101, 58, 32, 67, 77]...', original message: bytes can be at most 32766 in length; got 52634", "caused_by"=>{"type"=>"max_bytes_length_exceeded_exception", "reason"=>"max_bytes_length_exceeded_exception: bytes can be at most 32766 in length; got 52634"}}}}, :level=>:warn}

how can we increase 32766 limit for a single field ? or is there any workaround for such type of an issues??

Thanks
Shweta

[jsvd (João Duarte)]

This is a characteristic/protection on the elasticsearch side, for more information you can see a similar discuss topic: UTF8 encoding is longer than the max length 32766

[Shweta]

"index":no is the option we need to provide in template. But we are not sure about its syntax and where exactly to put it. Here is our template:

{
"order": 0,
"template": "delivery-app-",
"settings": {
"index": {
"refresh_interval": "5s"
}
},
"mappings": {
"default": {
"dynamic_templates": [
{
"template1": {
"mapping": {
"ignore_above": 1024,
"index": "not_analyzed",
"type": "{dynamic_type}",
"doc_values": true
},
"match": ""
}
}
],
"_all": {
"norms": {
"enabled": false
},
"enabled": true
},
"properties": {
"severity": {
"index": "not_analyzed",
"type": "string"
},
"offset": {
"type": "long",
"doc_values": "true"
},
"reportid": {
"index": "not_analyzed",
"type": "string"
},
"tmh_report": {
"index": "not_analyzed",
"type": "string"
},
"query": {
"index": "not_analyzed",
"type": "string"
},
"reportversion": {
"index": "not_analyzed",
"type": "string"
},
"emailid": {
"index": "not_analyzed",
"type": "string"
},
"source": {
"index": "not_analyzed",
"type": "string"
},
"thread": {
"index": "not_analyzed",
"type": "string"
},
"message": {
"index": "not_analyzed",
"type": "string"
},
"type": {
"index": "not_analyzed",
"type": "string"
},
"filter": {
"index": "not_analyzed",
"type": "string"
},
"profile_name": {
"index": "not_analyzed",
"type": "string"
},
"@timestamp": {
"type": "date"
},
"profileid": {
"index": "not_analyzed",
"type": "string"
},
"name": {
"index": "not_analyzed",
"type": "string"
},
"publish_date": {
"index": "not_analyzed",
"type": "string"
},
"status": {
"index": "not_analyzed",
"type": "string"
},
"timestamp": {
"index": "not_analyzed",
"type": "string"
}
}
}
},
"aliases": {}
}