Extract first 50 characters from message "body" into new field

Elastic Stack Logstash
1

Hello everyone,

Firstly, apologies for any formatting mistakes in this post - its my first one...

i'm relatively new to the Elastic stack and need to get a shortened version of the message body into another field so it can be used as a label within lens (I'm going to plump with the first 50 characters of the string).

Here's my logstash filter so far (this works as it simply copies the whole of "body" into a new field called "short_body" (so i know that there's nothing fundamentally wrong with my filter, i'm just not understanding the Ruby part).

`filter {
  if "guidewire" in [tags] {
    grok {
      match => {
        "message" => "(?<serverid>([a-z]{3}[0-9]{2}[a-z]{2}[0-9]{2}){0,1})\s+(?<username>([A-Za-z0-9.@\-]*){0,1})\s+(%{TIMESTAMP_ISO8601:logdate})?\s+(%{LOGLEVEL:loglevel})\s+(?<classname>([^\s]+){0,1})\s+%{GREEDYDATA:body}"
      }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
      tag_on_failure => [ "_not_interested" ]
    }
    ruby {
    code => 'shortbody = event.get("body")
             event.set("short_body",shortbody)'
    }
  if "_not_interested" in [tags] {
    drop { }
  }
  } else if "aggregator" in [tags] {
    grok {
      match => { "message" => "%{GREEDYDATA:body}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
  }
}`

The problem i have is that, whatever i do to try and substring the body field result in errors being written to the logstash log.

So far I've tried:

`code => 'shortbody = event.get("body")[0..50]
             event.set("short_body",shortbody)'`

which results in

Ruby exception occurred: undefined method `' for nil:NilClass

I tried many variations on this including using [0,50] as well as condensing the code to

code=> 'event.set("short_body", event.get("body")[0..50])'

Same problem

I also tried

`code => 'shortbody = event.get("body[0,50]")
             event.set("short_body",shortbody)'`

results in

Ruby exception occurred: Invalid FieldReference: body[0,50]

Can anyone offer any assistance , i thought i was trying to do something quite simple and I keep getting these errors thrown up and its driving me bananas.

Many thanks,
Adrian.

2

I don't know much of ruby, but you can do that using a simple grok filter.

Something like this:

 grok {
        match => { "message" => "(?<short_message>^.{0,50})" }
}

Testing this on my lab environment I get the following output.

{
         "@version" => "1",
             "host" => "elk",
    "short_message" => "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWX",
          "message" => "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ",
       "@timestamp" => 2020-10-19T17:14:07.417Z
}

My original message was 52 characters long, and my short_message is 50 characters long as you can see in the output.

In your case you would use your body and short_body fields.

3

I agree that grok is the way to go.

Regarding the ruby errors

Ruby exception occurred: undefined method `' for nil:NilClass

That is telling you that event.get("body") returned nil, meaning there is an event that does not have a [body] field.

Ruby exception occurred: Invalid FieldReference: body[0,50]

To reference a field as an array you would use [body][0, 50]

4

Will the grok filter work?

I already have a match filter that takes message and part of that message (the tail end) becomes the body field. It's this body field (not the whole message) i need to substring.

Great if it does, i'm just not sure how to get this to work.

5

Using another match filter on body did not seem to work. Logstash doesn't throw any log errors but there's no "short_body" field coming through:

match => {
        "message" => "(?<serverid>([a-z]{3}[0-9]{2}[a-z]{2}[0-9]{2}){0,1})\s+(?<username>([A-Za-z0-9.@\-]*){0,1})\s+(%{TIMESTAMP_ISO8601:logdate})?\s+(%{LOGLEVEL:loglevel})\s+(?<classname>([^\s]+){0,1})\s+%{GREEDYDATA:body}"
        "body" => "(?<short_body>^.{0,50})"
      }

Sorry if i'm not getting what you are saying ... any further ideas?

Many thanks,
Adrian.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.