Nested JSON parsing in dynamic field names

(Somya Makkad) #1

"0": {
"username": "rita",
"age": 21,
"occupation": "teacher"
"1": {
"username": "nita",
"age": 21,
"occupation": "police officer"
My log format looks as above. When applying the json filter it parses as 0.username, 0.age and so on. Can someone provide me the exact ruby code snippet so as to change the 0.username and all similar entries to just username i.e by applying the for loop in ruby filter.
Thanks in advance.

(Ry Biesemeyer) #2

Do you want to split the event into multiple events, each of which has a username, age, and occupation?

If so, you'll need to do a sequence of things, using the [@metadata] namespace (which is not exported from Logstash by default):

  • Use the JSON Filter to extract the object from the JSON string into [@metadata][string-indexed]
  • Convert the string-indexed object into an Array ([@metadata][items-array])
  • Split the Event on the resulting array, putting the result at [@metadata][single-item]
  • Do some cleanup to move the relevant sub-keys up to the top namespace.
filter {
  json {
    source => "message"
    target => "[@metadata][string-indexed]"
  ruby {
    code => "
      # extract the string-indexed object to a local variable
      string_indexed = event.get('[@metadata][string-indexed]')
      # iterate over the keys in the order of the string-encoded integer index
      items_array = string_indexed.keys.sort_by(&:to_i).map do |key|
        # get the item from the indexed map
      # set the items array.
      event.set('[@metadata][items-array]', items_array)
  split {
    field => "[@metadata][items-array]"
    target => "[@metadata][single-item]"
  mutate {
    rename => {
      "[@metadata][single-item][username]" => "username"
      "[@metadata][single-item][age]" => "age"
      "[@metadata][single-item][occupation]" => "occupation"

(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.