# How to aggregate conditionally logs

**URL:** <https://discuss.elastic.co/t/how-to-aggregate-conditionally-logs/342945>\
**Category:** Logstash\
**Created:** [September 13, 2023, 11:55am UTC](https://discuss.elastic.co/t/how-to-aggregate-conditionally-logs/342945 "2023-09-13T11:55:48Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Marieta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marieta/32/123249_2.png) [@Marieta](https://discuss.elastic.co/u/Marieta)\
**Post date:** [September 13, 2023, 11:55am UTC](https://discuss.elastic.co/t/how-to-aggregate-conditionally-logs/342945/1 "2023-09-13T11:55:48Z")

</div>

Hi I am trying to aggregate the following logs:

```auto
2023-09-06 07:36:22,573 | INFO | Thread-934 | Config | ENTERORDER: identifier = 'Barbie', buy = false, quantity = 290000.0, price = 96.1, account = '123', reference = '', customerId = '', originalBrokerId = 'ken'
2023-09-06 07:36:22,574 | INFO | Thread-934 | Config | processed: storeSchema = barcode, storeId = 'Barbie', currency = 'CHF', account = '123', quantity = 290000.0, allocAccounts = 
2023-09-06 07:36:22,574 | INFO | Thread-934 | Config | setOrigin = 'New'
2023-09-06 07:36:22,626 | INFO | Thread-934 | Config | glowServiceImpl::validateOrder::finally sw2 (49ms)
2023-09-06 07:36:22,630 | INFO | Thread-934 | Config | OrderUtils::validateOrder::setglowChecked value = 'Y:(55ms)'

```

To tags should be a one line and message concatenated. If the aggregate function is removed results are successfully processed however with Aggregate I cannot make it work, I tried multiple examples from logstash page but still no success. Many thanks in advance.

Here is my code:

```auto
input {
    pipeline {
        address => "logging-glow"
    }
}

filter {

    # Extract response_glow, appl_validation_time_ms, validation_interface_type, glow_roundtrip_ms, order_details, order_origin
    if ("ms" in [message]) or ("ENTERORDER" in [message]) or ("glowServiceImpl" in [message]) or ("setOrigin" in [message]){
        grok{
            match => {
                break_on_match => "true"
                "message" => [
                    ".*::%{GREEDYDATA:validation_interface_type}::.*'%{DATA:response_glow}:.*\(%{DATA:appl_validation_time_ms}\m",
                    "glowServiceImpl::%{GREEDYDATA:validation_interface_type}::.*\(%{DATA:glow_roundtrip_ms}\m",
                    "ENTERORDER: %{GREEDYDATA:order_details}",
                    ".*'%{GREEDYDATA:order_origin}'"
                ]
            }
        }
    }

    # Extract timestamp, severity, log_thread
    grok {
        match => {
            break_on_match => "false"
            "message" => ["%{TIMESTAMP_ISO8601:[@metadata][timestamp]}%{SPACE}\|%{SPACE}%{LOGLEVEL:severity}%{SPACE}\|%{SPACE}%{DATA:log_thread}%{SPACE}\|"]
        }
    }

    # Set timestamp
    date {
        match => ["[@metadata][timestamp]", "ISO8601"]
        target => "@timestamp"
        tag_on_failure => ["_dateparsefailure"]
    }

    # Set component to glow
    mutate {
        add_field => { "component" => "glow" }
    }

    # Set a unique document ID for each message to avoid duplicates
    fingerprint {
        source => ["@timestamp", "message"]
        target => "[@metadata][fingerprint]"
        concatenate_sources => true
    }
# Check if thread exists in memory if not create one, otherwise update. Keep max timestamp and aggregate tags of same thread every day
    aggregate {
        task_id => "%{thread_id}"
        timeout_task_id_field => "thread_id"
        timeout_timestamp_field => "@timestamp"
        code => "map['validation_interface_type'] = event.get('validation_interface_type')"
        push_previous_map_as_event => true
    }
}

output {
    file {
        path => "D:\APPL\logstash\Temp\logstash-logging-glow.log"
    }
}

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [September 13, 2023, 2:30pm UTC](https://discuss.elastic.co/t/how-to-aggregate-conditionally-logs/342945/2 "2023-09-13T14:30:07Z")

</div>

> [@Marieta](#):
>
> `task_id => "%{thread_id}"`

If thread\_id is not set then it always has the same value, so the aggregate will never flush the map because it never sees a different value for the thread\_id.

When you use push\_previous\_map\_as\_event the resulting event will only contain whatever fields you add to the map in the aggregate code option, plus @version and @timestamp.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 11, 2023, 2:30pm UTC](https://discuss.elastic.co/t/how-to-aggregate-conditionally-logs/342945/3 "2023-10-11T14:30:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.