Logstash - Parse formatted message with Ruby Code

Elastic Stack Logstash
1

Hi,

Thanks in advance for your time!
I have as input a line like the one below :

2019-12-18 08:28:36.810082121 string [number] logLevel string: tag1=val1|tag2=val2|tag3=val3|tag4=val4|tag5=val5|tag6=val6|

And I'd like my elasticsearch document to look like

{
  meaning_tag1=val1,
  meaning_tag3=val3,
  meaning_tag5=val5,
}

So it's basically a mapping from a tag number to its name + get rid of the tags I don't need

I end up so far with the below ruby code (my input line within ["message"]) :

ruby {
  code => '
     message_tag_value = event.get("message")
     tags_values = message_tag_value.split("|")
     tags_values.each do |tag_value|
       tag, value = tag_value.split("=")
      case tag
      when 1
        event.set("meaning_tag1", value)
      when 3
        event.set("meaning_tag3", value)
      when 5
        event.set("meaning_tag5", value)
      end
    end
' }

I'm not getting any error but the result isn't the one expecting (split seems to fail)
Hope I'm not so far from the correct code

Thanks!
Guillaume

2

The way I would personally do this is ......

First, split it into 2 fields with grok

2019-12-18 08:28:36.810082121 string [number] logLevel string:

tag1=val1|tag2=val2|tag3=val3|tag4=val4|tag5=val5|tag6=val6|

With something like:

%{TIMESTAMP} %{WORD:foo} [%{INT:bar}] %{WORD:loglevel} %{WORD:string}: %{NOTSPACE:kvtags}"

and then I would use the KV filter for the second part of this, something like this:

filter {
kv {
source => "kvtags"
field_split => "|"
value_split => "="
}
}

1 Like
3

Hi @gborg, I didn't know kv filter and it works just miles away better (and quicker I guess) ! Thanks

Below my config if it can help someone else :

grok {
  pattern_definitions => { "TIME1" => "%{YEAR}-%{MONTHNUM2}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND}" }
  match => { "message" => "%{TIME1:creationtime} %{GREEDYDATA:to-delete} incoming: %{GREEDYDATA:my-message}" }
}
kv {
  source => "my-message"
  allow_duplicate_values => false
  include_keys => [ "1", "3", "5" ]
  field_split => "|"
  value_split => "="
}
mutate {
  rename => { "1" => "meaning_tag1" }
  rename => { "3" => "meaning_tag3" }
  rename => { "5" => "meaning_tag5" }
  remove_field => ["to-delete-1", "fix-message" ]
}

Quick last question, if I want another datatype than String. My only solution is to add into my mutate filter a convert ? There is no other way either within kv filter or renaming/converting in one time ?
Asking this because I only found out recently that this is possible with grok filter using %{BASE10NUM:my-num:int}

Thanks a lot
Guillaume

4

Great that it's working well for you :slight_smile:

I don't think the KV filter has those options, since it is designed to take data regardless of order and pair it up as key:value

I looked over the documentation and don't see it there either

so mutate-convert sounds like the best solution to it

5

Agree,

Thanks again for checking out !

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.