Parse Json Array to flat object

Elastic Stack Logstash
1

Hello. I have a log string like this:

2017-08-18 14:26:07.757 [pool-18-thread-4] INFO r.s.c.i.s.w.WebSocketServerSessionHandler - SEND [63426612-2bbd-4e34-97c2-6bcc374fd654] OnItemAdded: {"item": {"id": "3ca48ea5-4dd2-47e3-9488-b6f850e5e7eb","type": "Voice","state": "ADDED","customer_id": "73015"},"properties": [{"name": "PRIMARY_DNIS","value": "12345"},{"name": "CALL_ID","value": "3ca48ea5-4dd2-47e3-9488-b6f850e5e7eb"},{"name": "UCID","value": "00001003951503055564"},{"name": "PRIMARY_ANI","value": "12345"},{"name": "AGENT_NUMBER","value": "12345"},{"name": "DNIS","value": "12345"},{"name": "AGENT_LOGIN","value": "12345"},{"name": "CALL_TYPE","value": "IN"},{"name": "CALLER_NUMBER","value": "12345"},{"name": "CREATETIME","value": "18.08.2017 14:26:07.610"},{"name": "ANI","value": "12345"},{"name": "EDUID","value": "00001003951503055564"},{"name": "TYPE","value": "VOICE"}],"participants": [{"name": "73015","type": "AGENT","id": "399149939","call_cause": "CAUSE_NORMAL","call_direction": "OUT","state": "INITIATING"},{"name": "73052","type": "AGENT","id": "1175032675","call_cause": "CAUSE_NORMAL","call_direction": "IN","state": "ADDED"}]}, correlation_id {}

I'am use this logstash config for parsing my logs:

filter {
    if [type] =~ "cis" {
    
        if [message] =~ "SEND" or [message] =~ "RECEIVED" {
                grok {
                    match => { 
                        "message" => "%{TIMESTAMP_ISO8601:timestamp}\s+\[%{GREEDYDATA:thread}\]\s+%{LOGLEVEL:loglevel}\s+%{GREEDYDATA:log_class}\s+\-\s+((?<operation>SEND|REQUEST)?\s+\[(?<guid>[0-9A-Fa-f]{8}[-][0-9A-Fa-f]{4}[-][0-9A-Fa-f]{4}[-][0-9A-Fa-f]{4}[-][0-9A-Fa-f]{12})\]\s+(%{DATA:event_type}):\s+(%{GREEDYDATA:json_data})\,\s+correlation_id)?%{GREEDYDATA:text}"                                 
                    }
                }
                date {
                    match => ["timestamp","YYYY-MM-dd HH:mm:ss.SSS"]
                    timezone => "Europe/Moscow"
                }
                json{
                    source => "json_data"
                }
                
               if [message] =~ "participants" {
                   
                    split{
                      field => "[json_data][participants]"
                      terminator => ","
                      target => "participant"
                      remove_field => [json_data]
                    }
                    mutate{
                     gsub => [ "participant", "\]", "" ]
                     gsub => [ "participant", "\[", "" ]
                    }
                    json{
                        source => "participant"
                    };
                }
        }
        else{
            drop { }
        }   
}    
}

How it work now, but partially.
My structrure
{"item": {"id": "3ca48ea5-4dd2-47e3-9488-b6f850e5e7eb","type": "Voice","state": "ADDED","customer_id": "73015"},

parsed as flat
json 4
json5
and is coorect for me, but this part (array) not transorming:

"participants": [{"name": "73015","type": "AGENT","id": "399149939","call_cause": "CAUSE_NORMAL","call_direction": "OUT","state": "INITIATING"},{"name": "73052","type": "AGENT","id": "1175032675","call_cause": "CAUSE_NORMAL","call_direction": "IN","state": "ADDED"}]

in table view:
json2
in json view:
json3
this fields available in list, but they empty:
json1

How i can do it part of my json data (array of participants) to flat objects like item.id ect?
How i can change my config?

2

Did you notice the _split_type_failure tag? It cannot find [json_data][participants]. Replace

           if [message] =~ "participants" {

                split{
                  field => "[json_data][participants]"
                  terminator => ","
                  target => "participant"
                  remove_field => [json_data]
                }
                mutate{
                 gsub => [ "participant", "\]", "" ]
                 gsub => [ "participant", "\[", "" ]
                }
                json{
                    source => "participant"
                }
            }

with

           if [message] =~ "participants" {
                split{
                  field => "[participants]"
                  target => "participant"
                  remove_field => [json_data]
                }
            }
1 Like
3

Great! It's work! But only first element of array move to field :frowning:

4

You should get two events. The first will have a participant item that only contains the first entry from the participants array, the second will have a participant item that only contains the second. Everything else will be duplicated in both events.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.