Parse Json Array to flat object

Hello. I have a log string like this:

2017-08-18 14:26:07.757 [pool-18-thread-4] INFO r.s.c.i.s.w.WebSocketServerSessionHandler - SEND [63426612-2bbd-4e34-97c2-6bcc374fd654] OnItemAdded: {"item": {"id": "3ca48ea5-4dd2-47e3-9488-b6f850e5e7eb","type": "Voice","state": "ADDED","customer_id": "73015"},"properties": [{"name": "PRIMARY_DNIS","value": "12345"},{"name": "CALL_ID","value": "3ca48ea5-4dd2-47e3-9488-b6f850e5e7eb"},{"name": "UCID","value": "00001003951503055564"},{"name": "PRIMARY_ANI","value": "12345"},{"name": "AGENT_NUMBER","value": "12345"},{"name": "DNIS","value": "12345"},{"name": "AGENT_LOGIN","value": "12345"},{"name": "CALL_TYPE","value": "IN"},{"name": "CALLER_NUMBER","value": "12345"},{"name": "CREATETIME","value": "18.08.2017 14:26:07.610"},{"name": "ANI","value": "12345"},{"name": "EDUID","value": "00001003951503055564"},{"name": "TYPE","value": "VOICE"}],"participants": [{"name": "73015","type": "AGENT","id": "399149939","call_cause": "CAUSE_NORMAL","call_direction": "OUT","state": "INITIATING"},{"name": "73052","type": "AGENT","id": "1175032675","call_cause": "CAUSE_NORMAL","call_direction": "IN","state": "ADDED"}]}, correlation_id {}

I'am use this logstash config for parsing my logs:

filter {
    if [type] =~ "cis" {
    
        if [message] =~ "SEND" or [message] =~ "RECEIVED" {
                grok {
                    match => { 
                        "message" => "%{TIMESTAMP_ISO8601:timestamp}\s+\[%{GREEDYDATA:thread}\]\s+%{LOGLEVEL:loglevel}\s+%{GREEDYDATA:log_class}\s+\-\s+((?<operation>SEND|REQUEST)?\s+\[(?<guid>[0-9A-Fa-f]{8}[-][0-9A-Fa-f]{4}[-][0-9A-Fa-f]{4}[-][0-9A-Fa-f]{4}[-][0-9A-Fa-f]{12})\]\s+(%{DATA:event_type}):\s+(%{GREEDYDATA:json_data})\,\s+correlation_id)?%{GREEDYDATA:text}"                                 
                    }
                }
                date {
                    match => ["timestamp","YYYY-MM-dd HH:mm:ss.SSS"]
                    timezone => "Europe/Moscow"
                }
                json{
                    source => "json_data"
                }
                
               if [message] =~ "participants" {
                   
                    split{
                      field => "[json_data][participants]"
                      terminator => ","
                      target => "participant"
                      remove_field => [json_data]
                    }
                    mutate{
                     gsub => [ "participant", "\]", "" ]
                     gsub => [ "participant", "\[", "" ]
                    }
                    json{
                        source => "participant"
                    };
                }
        }
        else{
            drop { }
        }   
}    
}

How it work now, but partially.
My structrure
{"item": {"id": "3ca48ea5-4dd2-47e3-9488-b6f850e5e7eb","type": "Voice","state": "ADDED","customer_id": "73015"},

parsed as flat
json 4
json5
and is coorect for me, but this part (array) not transorming:

"participants": [{"name": "73015","type": "AGENT","id": "399149939","call_cause": "CAUSE_NORMAL","call_direction": "OUT","state": "INITIATING"},{"name": "73052","type": "AGENT","id": "1175032675","call_cause": "CAUSE_NORMAL","call_direction": "IN","state": "ADDED"}]

in table view:
json2
in json view:
json3
this fields available in list, but they empty:
json1

How i can do it part of my json data (array of participants) to flat objects like item.id ect?
How i can change my config?

Did you notice the _split_type_failure tag? It cannot find [json_data][participants]. Replace

           if [message] =~ "participants" {

                split{
                  field => "[json_data][participants]"
                  terminator => ","
                  target => "participant"
                  remove_field => [json_data]
                }
                mutate{
                 gsub => [ "participant", "\]", "" ]
                 gsub => [ "participant", "\[", "" ]
                }
                json{
                    source => "participant"
                }
            }

with

           if [message] =~ "participants" {
                split{
                  field => "[participants]"
                  target => "participant"
                  remove_field => [json_data]
                }
            }
1 Like

Great! It's work! But only first element of array move to field :frowning:

You should get two events. The first will have a participant item that only contains the first entry from the participants array, the second will have a participant item that only contains the second. Everything else will be duplicated in both events.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.