Checking Field Length

Elastic Stack Logstash
1

I'm still relatively new to Logstash, and I'm looking to find a way to check the length of an error message within a filter. If the message exceeds a defined number of characters, I want to output to an alternate ES index, and if it's within the defined parameters, simply send it to our normal index. Is this possible, and if so, what is the best approach?

2

This is possible but you need to use a ruby filter. This untested snippet stores the desired index name in a new field:

filter {
  ruby {
    code => "
      event['indexname'] = event['somefield'].length > 10 ? 'long-fields' : 'short-fields'
    "
  }
}
3

Thanks for the assistance Magnus!

4

Sorry Magnus, but I'm just a little confused still as to how I define the index it will revert to. I assume
'long-fields' : 'short-fields' will accomplish this, with one of them defining the index
depending on how the condition resolves, but in my output to elasticsearch,
how is this accomplished?

output {
elasticsearch {
hosts => [ "my-hosts" ]
index => "What goes here-%{+YYYY-MM}" # Should I put "[event['indexname']]-%{+YYYY-MM}" here?
user => "user"
password => "pwd"
}
}

5

This is what you're looking for:

index => "%{indexname}-%{+YYYY-MM}"

You might want to store the index name in a @metadata field instead so it doesn't end up in the event sent to Elasticsearch.

See https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#logstash-config-field-references.

6

Thanks again!

7

How can I trim all the fields in message based on a particular length? Lets say I have a very huge stack trace.
So if I have a stack trace like aabcd..... , I want to take the first 30k characters of a field and truncate the remaining characters,
If I want to index that in ES it has a limit of 32766 for storing it as a string.

8

@rakesh, please start a new thread instead of resurrecting this very old one.