Parsing of a sub-element

Elastic Stack Logstash
1

Hi,

I've got an input like this:
{
"_index": "index-2019.07.11",
"_type": "doc",
"_id": "id",
"_version": 1,
"_score": null,
"_source": {
"message": "{ "time": "2019-07-11T14:17:18.7132501+03:00", "level": "ERROR", "message": "{\"AddedTime\":\"2019-07-11 14:17:18.0570\",\"RequestedTime\":\"2019-07-11 14:17:18.0570\",..."" }\r",
"path": "/mnt/logs/CachedTariffProvider_Error.log",
"host": "host",
"@version": "1",
"@timestamp": "2019-07-11T13:30:54.756Z"
},

The main JSON container consist of internal field "message", which also has got a field named "message".

I have to extract all the fields from those two fields "message" and put them at the top level in order to have the final output look like:
index
type
id
version
_score
time
level
AddedTime
RequestedTime
path
host
@version
@timestamp
...

Additional difficulty is that there are special symbols in the field message: "/", """, ":", etc.

I have spent a lot of time searching this and other forums, as well as reading the official docs, and found no clear answer (.

My logstash filter config:

filter {
date {
match => [ "time", "yyyy-MM-dd HH:mm:ss.SSSS" ]
target => "@timestamp"
}

    json {
            source => "message"
    }

    split {
            field => "message"
    }

    mutate {
            add_field {
                    "time" => "%{[message][time]}"
                    "level" => "%{[message][level]}"
                    "message" => "%{[message][message]}"
                    "logger" => "%{[message][logger]}"
                    "exception" => "%{[message][exception]}"
                    "EventId.Id" => "%{[message][EventId.Id]}"
                    "EventId.Name" => "%{[message][EventId.Name]}"
                    "EventId" => "%{[message][EventId]}"
            }

            remove_field => [ "[message]" ]
    }

}

An errors happen:

  1. Can't merge a non object mapping [EventId] with an object mapping [EventId]...
  2. Expected one of #, => at line 23, column 13 (byte 404) after filter ...
    ...and some other ones.

How could I extract all the fields from "messages" and put them at top level of JSON output?

Any help would be appreciated, thank you!

2

What is the complete error message and what does line 23 look like?

3

I've already fixed it, missed out "=>" between "add_field" and "{"

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.