How to extract JSON content in message to top level

Elastic Stack Logstash
1

Sorry if it is a duplicated post.
I have been searching google and the forum for answer, however I still could not find it.
It would be nice if someone can direct me to the answer if it is duplicated.

I am currently using Logstash to receive some of my logs via UDP and save it to the file.
The content sent to Logstash is as follow:

{
"key1":"value1",
"key2":"value2",
"key3":"value3",
"key4":"value4"
}

and in my output file looks like:

{
"message": {
"key1":"value1",
"key2":"value2",
"key3":"value3",
"key4":"value4"
},
"@timestamp":"....."
}

desired output file:

{
"key1":"value1",
"key2":"value2",
"key3":"value3",
"key4":"value4",
"@timestamp":"....."
}

I am hoping to extract all key and value to the top level rather than storing in message.
I am aware I can use mutate.addfield but that means if I have 50 key, my add_field would have 50 entry.

mutate {
add_field => {
"key1" => "%{[message][key1]}"
}
}

I am looking for a better solution to it, so that all key in message can be extracted to the top level.
I am currently using Logstash 6.4.2

From what I have now in my filter without add_field:

filter {
json {
source => "message"
target => "message"
}
mutate {
remove_field => ["tags","@version","host"]
}
}

2

What does your current config look like given that you end up with the keys under message?

3

My current full config is like this:

input {
udp {
port => 6578
}
}

filter {
json {
source => "message"
target => "message"
}
mutate {
remove_field => ["tags","@version","host"]
}
}

output {
file {
codec => json_lines
path => "/data/parse.log"
}
}

4

Remove the target parameter in the JSON filter. That should put extracted fields at the root level.

5

I think logstash just stop writing out data received. I dont see error messages from logstash log.

tail /var/log/logstash/logstash-plain.log

6

bump

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.