Dynamically create fields from another field with json input file

Elastic Stack Logstash
1

Hi,
I have json input that contains appname with their version and other field with standard string like :

{
  "app1": "1.2.3",
  "app2": "3.2.1",
  "arch": "virtual"
}

I would like to be able to use filter versions in kibana like "app1<1.2.3" but I wasn't able to find how to do that.

So if thought maybe creating new fields like appname1_major, appname1_minor and appname1_patch.
But I cant manage to do it in logstash
I have no way of knowing wich appname will be in the file, the version is always in the same format : 1.2.3

I managed to do it for only one field with grok but I can't do it fo every field that might appear or not

grok { 
  match => { "app1" => "%{INT:app1_major}\.%{INT:app1_minor}\.%{INT:app1_patch}" } 
}

This is ugly and takes more storage space but that's all I have for now
Is there a good or better way to do this ?

Thanks

2

Welcome!

IMHO you are doing it the right way.
I'd probably do something like:

{
 "app1": {
  "major": 1,
  "minor": 2,
  "patch": 3
 }
}

instead of:

{
  "app1_major": 1,
  "app1_minor": 2,
  "app1_patch": 3
}

But that's a matter of taste :wink:

3

yeah I agree but I would like to keep the full version too, it is nicer and way easier in a dashboard

Anyway, maybe my post wasn't clear enough but I don't know IF app1 exist or even what would really be app1's name

I made a regex to match a key value pair

/(?:\"|\')(?<key>[^"]*)(?:\"|\')(?=:)(?:\:\s*)(?:\"|\')?(?<major>\d+)\.(?<minor>\d+)\.(?<patch>\d+)/gm

This regex match the key name and each integer in version
But I dont know how to tell logstash to create fields => value whit this regex
like ${key}_major => ${major}

4

I see. I don't know if this can be done automatically without using the ruby filter.

But actually I just realized that Elasticsearch supports as well the version type, so basically you don't need to do all that I guess...

5

For that I need to specify each field statically, which I can't do

6

You could try something like

    ruby {
        code => '
            event.to_hash.each { |k, v|
                if v.is_a? String
                    matches = /^(\d+)\.(\d+)\.(\d+)$/.match(v)
                    if matches
                        newKey = k.gsub(/ /, "_").gsub(/[\[\]]/, "")
                        event.set("[apps][#{newKey}]", {
                            "major" => matches[1].to_i,
                            "minor" => matches[2].to_i,
                            "patch" => matches[3].to_i
                        })
                        # Alternatively
                        #event.set("#{newKey}_major", matches[1].to_i)
                        #event.set("#{newKey}_minor", matches[2].to_i)
                        #event.set("#{newKey}_patch", matches[3].to_i)
                    end
                end
            }
        '
    }
1 Like
7

But do you have a prefix or something? In which case a dynamic template would work.

8

I actually ended up with something like this (but yours is better)

ruby {
      code => "
        event.to_hash.each do |key, value|
        if value =~ /^(?<major>\d+)\.(?<minor>\d+)\.(?<patch>\d+)$/
          event.remove(key)
          event.set(key + '.full', value)
          event.set(key + '.major', $~[:major])
          event.set(key + '.minor', $~[:minor])
          event.set(key + '.patch', $~[:patch])
        end
      end
      "
      }

I still need to play with the remove event to remove the key or else I am not able to create key.major....
But this is working as I want and easy enough to understand
@dadoonet no prefix, just "randomappname": "1.2.3"

Thanks all for your help

9

I see. So if the value matches the regex, I'd just set the value to a field named version_[key] and use a dynamic template which matches any field where name starts with version_...

My 2 cents.