Hi,
I've got an input like this:
{
"_index": "index-2019.07.11",
"_type": "doc",
"_id": "id",
"_version": 1,
"_score": null,
"_source": {
"message": "{ "time": "2019-07-11T14:17:18.7132501+03:00", "level": "ERROR", "message": "{\"AddedTime\":\"2019-07-11 14:17:18.0570\",\"RequestedTime\":\"2019-07-11 14:17:18.0570\",..."" }\r",
"path": "/mnt/logs/CachedTariffProvider_Error.log",
"host": "host",
"@version": "1",
"@timestamp": "2019-07-11T13:30:54.756Z"
},
The main JSON container consist of internal field "message", which also has got a field named "message".
I have to extract all the fields from those two fields "message" and put them at the top level in order to have the final output look like:
index
type
id
version
_score
time
level
AddedTime
RequestedTime
path
host
@version
@timestamp
...
Additional difficulty is that there are special symbols in the field message: "/", """, ":", etc.
I have spent a lot of time searching this and other forums, as well as reading the official docs, and found no clear answer (.
My logstash filter config:
filter {
date {
match => [ "time", "yyyy-MM-dd HH:mm:ss.SSSS" ]
target => "@timestamp"
}
json {
source => "message"
}
split {
field => "message"
}
mutate {
add_field {
"time" => "%{[message][time]}"
"level" => "%{[message][level]}"
"message" => "%{[message][message]}"
"logger" => "%{[message][logger]}"
"exception" => "%{[message][exception]}"
"EventId.Id" => "%{[message][EventId.Id]}"
"EventId.Name" => "%{[message][EventId.Name]}"
"EventId" => "%{[message][EventId]}"
}
remove_field => [ "[message]" ]
}
}
An errors happen:
- Can't merge a non object mapping [EventId] with an object mapping [EventId]...
- Expected one of #, => at line 23, column 13 (byte 404) after filter ...
...and some other ones.
How could I extract all the fields from "messages" and put them at top level of JSON output?
Any help would be appreciated, thank you!