Extract specific fields from Json using Logstash

I have this json file

{
  "type": "resource",
  "headers": {
    "destination_channel": "data"
  },
  "body": {
    "type":"resource",
    "resourceId":"estimatorB",
    "resourceType":null,
    "resourceValue":"data.csv",
    "resourceSettings":{
    },
    "resourceContext":{ 
    },
    "p_id":"123",
    "b_id":"block_789"
  }
}


and I need to take only three fields and push them to Elasticsearch, which they're:

resourceValue
p_id
b_id

here is the code I'm using

input {
  file {
    path => "/usr/share/input/test.json"
    start_position => beginning
    sincedb_path => "/dev/null"
   
  }
}

filter {
  json {
    source => "message"    
  }
  ruby {
    code => '
        arrayOfEvents = Array.new()
        ts = event.get("[body]")
        ts.each do |k,v|
          if k == "resourceValue"
            arrayOfEvents.push(data)
          elsif k == "pipelineId"
            arrayOfEvents.push(data)
          elsif k == "blockId"
            arrayOfEvents.push(data)
          end                        
        end
        arrayOfEvents.push(data)
        event.set("event",arrayOfEvents)        
    '
  }
  split { field => "event" }
}


output {
  stdout {}
}

but it throws an exception ruby split error !

What are you using "data" to refer to in the ruby filter, and what are you trying to do with the .push immediately before the event.set?

@Badger I thought about it this way:
I get the fields, or iterate through them -> read the targeted field value and then push it to a temp array -> return it.
to get this output:

{
 "resourceValue":"data.csv",
 "p_id":"123",
 "b_id":"block_789"
}

data is nil, so the event array is nil, which is not splittable. If you change the first three occurrences of data to v, and change blockId to b_id, and change pipelineId to p_id, and delete the fourth push you will get three events

{
"event" => "123",
"@timestamp" => 2019-09-09T17:41:31.316Z
}
{
"event" => "data.csv",
"@timestamp" => 2019-09-09T17:41:31.316Z
}
{
"event" => "block_789",
"@timestamp" => 2019-09-09T17:41:31.316Z
}

which I do not think is what you want. I do not think you need to use ruby at all.

    mutate {
        add_field => {
            "resourceValue" => "%{[body][resourceValue]}"
            "p_id" => "%{[body][p_id]}"
            "b_id" => "%{[body][b_id]}"
        }
    }
    mutate { remove_field => [ "body", "headers", "message", "type" ] }

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.