Parsing incoming Json Data

Elastic Stack Logstash
1

Hi,

I have following incoming data
{"event":[{"duration":"3","time":"2019-02-04 17:49:27","event":"achievementUnlock","message":"Achievement unlocked"},{"duration":"5","time":"2019-02-04 17:49:27","event":"achievementUnlock","message":"Achievement unlocked"},{"duration":"6","time":"2019-02-04 17:49:27","event":"achievementUnlock","message":"Achievement unlocked"}]}

Need to push each element of event array as a separate entry to elastic
one document will look like this
hits:[
"_source": {
"@timestamp": "2019-02-04T09:44:20.379Z",
"duration":"3"
"time":2019-02-04
"event": achievementUnlock
"message":Achievement unlocked,
"@version": "1"
},
{
"_source": {
"@timestamp": "2019-02-04T09:44:20.379Z",
"duration":"5"
"time":2019-02-04
"event": achievementUnlock
"message":Achievement unlocked,
"@version": "1"
}
}
]

instead I am getting it in below format

"hits": [
{

"_source": {
"event": {
"time": "2019-02-04 17:43:29",
"event": "achievementUnlock",
"duration": "3",
"message": "Achievement unlocked"
},
"@version": "1",
"@timestamp": "2019-02-04T12:13:29.957Z"
}
},
{

"_source": {
"event": {
"time": "2019-02-04 17:43:29",
"event": "achievementUnlock",
"duration": "5",
"message": "Achievement unlocked"
},
"@version": "1",
"@timestamp": "2019-02-04T12:13:29.957Z"
}
}

logstash conf I am using is below

input{
rabbitmq {
host => "localhost"
queue => "logger_queues"
durable =>false
codec=>json

}

}
filter{
split {
field => "event"
}

date {
		match => [ "time", "YYYY-MM-dd HH:mm:ss" ]
		target => "@timestamp"
       
       
	}

}
output{
elasticsearch{
hosts => "localhost"
index => "test"
document_type => "test"
}
}

2

This is what the split filter does. The extracted fields will not be in the root of the doc, they will be in the event field.
To move them into the root, add this to your split filter setting section:

  add_field => {
    "duration" => "%{[event][duration]}"
    "event" => "%{[event][event]}"
    # add all other fields similarly
  }
  remove_field => ["event"]
3

Thanks... I got it working by below tweak to my incoming json.
I made it as only a list of Json objects and used codec => json.
And that did the trick

4

Nice to know.

Yes, the JSON codec will break up an array of JSON objects into individual docs (events).

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.