Logstash: manipulate json data

[nameisnotimportant]

Hi,

i retrieved data from ES and would like to add/modify the json data before index it.

My json data looks like below:

{
  "john" : {
    "enabled" : true,
    "roles" : [
      "developer"
    ],
    "rules" : {
      "any" : [
        {
          "field" : {
            "dn" : "CN=john,OU=IT,OU=MY,DC=DomainName,DC=local"
          }
        }
      ]
    },
    "metadata" : { }
  },
  "shyap" : {
    "enabled" : true,
    "roles" : [
      "superuser",
      "reportinguser"
    ],
    "rules" : {
      "any" : [
        {
          "field" : {
            "dn" : "CN=shyap,OU=IT,OU=MY,DC=DomainName,DC=local"
          }
        }
      ]
    },
    "metadata" : { }
  }
}

My config file:

input {
  http_poller {
      urls => {
          test1 => { 
              method => get
              user => "elastic"
              password => "changeme"
              url => "https://hostname:9200/_security/role_mapping"
              headers => {
                  Accept => "application/json"
              }
          }
      }    
  request_timeout => 60
  schedule => {"every" => "2s"}
  codec => "json"
  cacert => "/etc/elasticsearch/certs/root2016.crt"
  }
}
filter {
  json {
    source => "message"
  }
  mutate {
    remove_field => ["@timestamp"]
    remove_field => ["@version"]
    remove_field => ["message"]
  }
}
output {
  stdout { codec => rubydebug }
}

This is the role mapping data.

I would like to know how do i transform the data into following:

First step i am trying is to add new field 'userid' and give the value of 'shyap' and 'john'.

When i put in the 'add_field' option, nothing happne.

i use following json filter:

json {
    source => "message"
    add_field => {"userid" => "%{[message][0]}"}
  }

However, if i use mutate filter, i get only one userid field added. Somehow, the two records were treated as one?

{
     "shyap" => {
         "enabled" => true,
           "roles" => [
            [0] "superuser",
            [1] "reportinguser"
        ],
           "rules" => {
            "any" => [
                [0] {
                    "field" => {
                        "dn" => "CN=shyap,OU=IT,OU=MY,DC=DomainName,DC=local"
                    }
                }
            ]
        },
        "metadata" => {}
    },
    "userid" => "%{[message][0]}",
      "john" => {
         "enabled" => true,
           "roles" => [
            [0] "developer"
        ],
           "rules" => {
            "any" => [
                [0] {
                    "field" => {
                        "dn" => "CN=john,OU=IT,OU=MY,DC=DomainName,DC=local"
                    }
                }
            ]
        },
        "metadata" => {}
    }
}

Can someone guide me how do i proceed ?

Thanks in advance.

[Wolfram_Haussig]

Hi,

After parsing the JSON you have to use the split filter. With this, you can split the single document into separate documents for each user. After this, your mutate filter should work.

Best regards
Wolfram

[nameisnotimportant]

i tried that with following:

split {
    field => "message"
  }

and i got error:

[WARN ] 2020-11-04 15:21:37.137 [[main]>worker18] split - Only String and Array types are splittable. field:message is of type = NilClass

Either the json filter not working or the codec not working...i am yet to figure it out.

[Wolfram_Haussig]

I am sorry, I didn't see that the json is a hash instead of an array.

I fear that you have to convert that hash/dictionary to an array first using ruby. After that you can use the split filter to convert this array to separate documents.

Maybe another user has a better idea?

[Badger]

I would start with

filter {
    ruby {
        code => '
            a = []
            event.get("data").each { |k, v|
                a << v.merge( { "user" => k } )
            }
            event.set("users", a)
        '
        remove_field => [ "data" ]
    }
    split { field => "users" }
    mutate { join => { "[users][roles]" => "," } }
}

then you just need to move fields around using mutate.

[nameisnotimportant]

Thank you Wolfram and Badger.
With your help, i am almost done. What is pending now is the "dn" key value pair. I am still not able to put it at the same level as role.

This is my latest logstash config file:

input {
  http_poller {
      urls => {
          test1 => { 
              method => get
              user => "elastic"
              password => "changeme"
              url => "https://hostname:9200/_security/role_mapping"
              headers => {
                  Accept => "application/json"
              }
          }
      }    
  request_timeout => 60
  schedule => {"every" => "1s"}
  codec => "plain"
  cacert => "/etc/elasticsearch/certs/root2016.crt"
  }
}

filter {
  ruby {
    code => '
         require "date"
         #add new field to apply to alias with other system
         a = []
         b = {"custom_string_field" => "nil"}
         c = {"custom_date_field" => DateTime.now.iso8601(3)}

         JSON.parse(event.get("message")).each { |k, v|
           v.merge!(b)
           v.merge!(c)
           a << v.merge( { "userid" => k } )
         }
         event.set("users", a)
    '
    remove_field => [ "message" ]
  }

  split {
    field => "users"
  }
  mutate {
    join => { "[users][roles]" => "," }
  }

  mutate {
    remove_field => ["@timestamp"]
    remove_field => ["@version"]
  }

  #to remove root
  ruby {
    code => '
      event.get("users").each { |k, v|
        event.set(k, v)
      }
      event.remove("users")

      event.get("rules").each { |k, v|
        event.set(k, v)
      }
      event.remove("rules")

     # the following commented part doesn't work
     #event.get("any").each { |k, v|
     #   event.set(k, v)
     # }
     # event.remove("any")
    '
  }
}

output {
  stdout { codec => rubydebug }
}

[Badger]

[any] is an array. You can move the dn to the root level using

mutate { rename => { "[any][0][field][dn]" => "dn" } }

[nameisnotimportant]

Thanks Badger. That did the tricks !
Although i still wonder around, what if i have more than one items in the array ? But, i prefer to try it out first before i ask for help.

Thanks a lot.
Regards.