Problem with multiple aggregate function

[Alankrit_Aman_Mishra]

Hello,
I have logs that have a similar format.
INFO - 98765 - http_Method - POST
WARN - 12345 - latency - 55
ERROR - 12345 - status_code - 200
WARN - 98765 - latency - 99
XYZ - 12345 - country - XYZ
ERROR - 98765 - status_code - 404
INFO - 12345 - http_Method - GET
XYZ - 98765 - country - ABC

in short, logs may come in any sequence. we only have four fields to care about.
We need to copy and paste data from the different fields to the logs file which contains the country field.

Expected Output:
12345-http_Method-GET-country-XYZ-status_code-200-latency-55
98765-http_Method-POST-country-ABC-status_code-404-latency-99

We are using 3 aggregate functions.

First aggregate function.
if [level] == "INFO" {
aggregate {
task_id => "%{api_id}"
code => "map['status_code'] ||= 0 ; map['status_code'] += event.get('status_code')"
}

if [level] == "XYZ" {
aggregate {
task_id => "%{api_id}"
code => "event.set('status_code', map['status_code'])"
}

Second aggregate function

if [level] == "WARN " {
aggregate {
task_id => "%{api_id}"
code => "map['latency'] ||= 0 ; map['latency'] += event.get('latency')"
}

if [level] == "XYZ" {
aggregate {
task_id => "%{api_id}"
code => "event.set('latency', map['latency'])"
}

Third aggregate function

if [level] == "ERROR " {
aggregate {
task_id => "%{api_id}"
code => "map['status_code'] ||= 0 ; map['status_code'] += event.get('status_code')"
}

if [level] == "XYZ" {
aggregate {
task_id => "%{api_id}"
code => "event.set('status_code', map['status_code'])"
}

The issue is that in the output some times, all values are present, but many times all three values are not present.

Anyone has any idea or any kind of approach please let us know.

[Badger]

I would suggest something like

    dissect { mapping => { "message" => "%{level} - %{api_id} - %{key} - %{value}" } }
    aggregate {
        task_id => "%{api_id}"
        code => '
            case event.get("key")
            when "http_Method"
                map["method"] = event.get("value")
            when "latency"
                map["latency"] = event.get("value").to_i
            when "status_code"
                map["status"] = event.get("value").to_i
            when "country"
                map["country"] = event.get("value")
            end
            event.cancel
        '
        push_map_as_event_on_timeout => true
        timeout_task_id_field => "api_id"
        timeout => 6
    }

If you require a specific output format you could rearrange the fields on the event in the timeout_code option.