Logstash split one JSON document into multiple events

Elastic Stack Logstash
1

Would you please help split a single JSON document being sent to Logstash into multiple events?

There are some fields that are shared across the events but the fields that have a trailing number, such as "_0001", are unique to that numbered event. In the below example, there happen to be twenty events in the single document, but it could be anywhere between one event to potentially thousands.

For extra credit, an extra nice solution wouldn't require me spelling out every single field but would instead automate the splitting based on whether or not there is that trailing event number "_0001". This very-nice-to-have would help address that there are several hundred fields that may or may not be present in any given document. For example, the following document doesn't mention SMF123_UDS_USER_NAME_0001, but some other documents do.

{
"MFSOURCETYPE": "SMF123",
"SMF123_SERVER_CONFIG_DIR": "/var/zosconnect/servers/provider/",
"SMF123_SERVER_JOBNAME": "ZC0001",
"SMF123_SERVER_SYSTEM": "ZOS1",

"SMF123S1_API_NAME_0001": "ZOS Connect Policy Inquiry",
"SMF123S1_API_NAME_0002": "ZOS Connect Policy Inquiry",
"SMF123S1_API_NAME_0003": "ZOS Connect Policy Inquiry",
...
"SMF123S1_API_NAME_0020": "ZOS Connect Policy Inquiry",

"SMF123S1_CLIENT_IP_ADDR_0001": "1.2.3.4",
"SMF123S1_CLIENT_IP_ADDR_0002": "1.2.3.4",
"SMF123S1_CLIENT_IP_ADDR_0003": "5.6.7.8",
...
"SMF123S1_CLIENT_IP_ADDR_0020": "9.1.2.3",

"SMF123S1_HTTP_RESP_CODE_0001": 200,
"SMF123S1_HTTP_RESP_CODE_0002": 204,
"SMF123S1_HTTP_RESP_CODE_0003": 204,
...
"SMF123S1_HTTP_RESP_CODE_0020": 200,

"SMF123S1_REQ_HDR1_0001": "ABCD",
"SMF123S1_REQ_HDR1_0002": "",
"SMF123S1_REQ_HDR1_0003": "",
...
"SMF123S1_REQ_HDR1_0020": "",

"SMF123S1_REQ_HDR2_0001": "EFG",
"SMF123S1_REQ_HDR2_0002": "",
"SMF123S1_REQ_HDR2_0003": "XYZ",
...
"SMF123S1_REQ_HDR2_0020": "",

"SMF123S1_REQ_HDR3_0001": "HIJK",
"SMF123S1_REQ_HDR3_0002": "",
"SMF123S1_REQ_HDR3_0003": "",
...
"SMF123S1_REQ_HDR3_0020": "",

"SMF123S1_REQ_HDR99_0001": "LMNOP",
"SMF123S1_REQ_HDR99_0002": "",
"SMF123S1_REQ_HDR99_0003": "",
...
"SMF123S1_REQ_HDR99_0020": ""
}

Desired state would look like:

split out event _0001:

{
"MFSOURCETYPE": "SMF123",
"SMF123_SERVER_CONFIG_DIR": "/var/zosconnect/servers/provider/",
"SMF123_SERVER_JOBNAME": "ZC0001",
"SMF123_SERVER_SYSTEM": "ZOS1",

"SMF123S1_API_NAME": "ZOS Connect Policy Inquiry",
"SMF123S1_CLIENT_IP_ADDR": "1.2.3.4",
"SMF123S1_HTTP_RESP_CODE": 200,
"SMF123S1_REQ_HDR1": "ABCD",
"SMF123S1_REQ_HDR2": "EFG",
"SMF123S1_REQ_HDR3": "HIJK",
...
"SMF123S1_REQ_HDR99": "LMNOP",
}

split out event _0002:

{
"MFSOURCETYPE": "SMF123",
"SMF123_SERVER_CONFIG_DIR": "/var/zosconnect/servers/provider/",
"SMF123_SERVER_JOBNAME": "ZC0001",
"SMF123_SERVER_SYSTEM": "ZOS1",

"SMF123S1_API_NAME": "ZOS Connect Policy Inquiry",
"SMF123S1_CLIENT_IP_ADDR": "1.2.3.4",
"SMF123S1_HTTP_RESP_CODE": 204,
"SMF123S1_REQ_HDR1": "",
"SMF123S1_REQ_HDR2": "",
"SMF123S1_REQ_HDR3": "",
...
"SMF123S1_REQ_HDR99": "",
}

split out event _0003:

{
"MFSOURCETYPE": "SMF123",
"SMF123_SERVER_CONFIG_DIR": "/var/zosconnect/servers/provider/",
"SMF123_SERVER_JOBNAME": "ZC0001",
"SMF123_SERVER_SYSTEM": "ZOS1",

"SMF123S1_API_NAME": "ZOS Connect Policy Inquiry",
"SMF123S1_CLIENT_IP_ADDR": "5.6.7.8",
"SMF123S1_HTTP_RESP_CODE": 204,
"SMF123S1_REQ_HDR1": "",
"SMF123S1_REQ_HDR2": "XYZ",
"SMF123S1_REQ_HDR3": "",
...
"SMF123S1_REQ_HDR99": "",

}

...

split out event _0020:

{
"MFSOURCETYPE": "SMF123",
"SMF123_SERVER_CONFIG_DIR": "/var/zosconnect/servers/provider/",
"SMF123_SERVER_JOBNAME": "ZC0001",
"SMF123_SERVER_SYSTEM": "ZOS1",

"SMF123S1_API_NAME": "ZOS Connect Policy Inquiry",
"SMF123S1_CLIENT_IP_ADDR": "9.1.2.3",
"SMF123S1_HTTP_RESP_CODE": 200,
"SMF123S1_REQ_HDR1": "",
"SMF123S1_REQ_HDR2": "",
"SMF123S1_REQ_HDR3": "",
...
"SMF123S1_REQ_HDR99": ""

}

Thank you very much!

2

It only took 20 minutes to create this:

    json { source => "message" target => "[@metadata][json]" remove_field => [ "message" ] }
    ruby {
        code => '
            def number_or_nil(string)
                Integer(string || "")
            rescue ArgumentError
                nil
            end

            a = []
            common = {}
            event.remove("[@metadata][json]").each { |k, v|
                n = k.match(/(.*)_(\d{4})$/)
                if n
                    i = number_or_nil(n[2])
                    a[i] ||= {}
                    a[i][n[1]] = v
                else
                    common[k] = v
                end
            }

            a.each_index { |i|
                if a[i]
                    a[i].merge(common)
                end
            }
            event.set("[@metadata][data]", a)
        '
    }

    split { field => "[@metadata][data]" }

    ruby {
        code => '
            event.remove("[@metadata][data]").each { |k, v|
                event.set(k, v)
            }
        '
    }
1 Like