Any way to limit field length?

Elastic Stack Logstash
1

Hi,

I sometimes receive messages that are huge... (719939 bytes) and indexation to elasticsearch failed with a max_bytes_length_exceeded_exception (max limit is 32766 bytes).
I want to detect those messages in kibana, so I would like to do with logstash something like that:

if len(message) > 10000 {
message = message[0:10000] // get only the first bytes
set tags => ["long message"]
}

However I didn't find that kind of filter.
Did I miss something?

Regards.

1 Like
2

I don't believe there is such a filter, but it would be easy to write a custom ruby filter. Something like

ruby {
  code => "
    event['message'] = event['message'][0..9999] if event['message'].length > 10000
    event.tag 'long message'
  "
}

should work. You can probably drop the conditional:

ruby {
  code => "
    event['message'] = event['message'][0..9999]
    event.tag 'long message'
  "
}
1 Like
3

You can do this in ES via the mappings too, see https://www.elastic.co/guide/en/elasticsearch/reference/2.3/ignore-above.html

4

Wow, thanks you two.
This is very helpful.

5

See Truncate and Range filters

2 Likes