# Logstash field is never shown after aggregation **URL:** \ **Category:** Logstash\ **Created:** [December 13, 2024, 3:21pm UTC](https://discuss.elastic.co/t/logstash-field-is-never-shown-after-aggregation/371963 "2024-12-13T15:21:25Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Hannah\_J\_Swystun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hannah_j_swystun/32/139547_2.png) [@Hannah\_J\_Swystun](https://discuss.elastic.co/u/Hannah_J_Swystun)\ **Post date:** [December 13, 2024, 3:21pm UTC](https://discuss.elastic.co/t/logstash-field-is-never-shown-after-aggregation/371963/1 "2024-12-13T15:21:25Z")
I have logstash version 7.8.0 Can someone tell me why the aggregation below never shown THREAD\_ID field into documents please ? My field : thread\_id is added in the end of aggregation .. ```auto 2024-12-14 12:00:01 thread-1 SOAP message <> 2024-12-14 12:00:02 thread-1 SOAP message >>Envelope 012<< 2024-12-14 12:05:03 thread-2 SOAP message <> 2024-12-14 12:05:04 thread-2 SOAP message >>Envelope 022<< ``` ```auto filter { grok { match => { "message" => [ '%{TIMESTAMP_ISO8601:log_timestamp} thread-%{INT:thread_id} SOAP message <<(?.*?)>>', '%{TIMESTAMP_ISO8601:log_timestamp} thread-%{INT:thread_id} SOAP message >>(?.*?)<<' ] } } aggregate { task_id => "%{thread_id}" code => " map['soap_in'] ||= [] map['soap_out'] ||= [] # Capture soap_in with timestamp if exists if event.get('soap_in') map['soap_in'] << {'soap_in' => event.get('soap_in'), 'log_timestamp' => event.get('log_timestamp')} end # Capture soap_out with timestamp if exists if event.get('soap_out') map['soap_out'] << {'soap_out' => event.get('soap_out'), 'log_timestamp' => event.get('log_timestamp')} end # Once both soap_in and soap_out are available, emit the aggregated event if map['soap_in'] && map['soap_out'] event.set('soap_in', map['soap_in']) event.set('soap_out', map['soap_out']) event.set('thread_id', event.get('thread_id')) event.cancel() end " push_previous_map_as_event => true timeout => 3 } mutate { remove_field => ["message"] } } ``` Result never shown thread\_id ```auto { "took" : 0, "timed_out" : false, "_shards" : { "total" : 1, "successful" : 1, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 2, "relation" : "eq" }, "max_score" : 1.0, "hits" : [ { "_index" : "app-aggregate-2024.12.13", "_type" : "_doc", "_id" : "qop5wJMB81mNBoMqWzzC", "_score" : 1.0, "_source" : { "@version" : "1", "soap_out" : [ { "log_timestamp" : "2024-12-14 12:00:02", "soap_out" : "Envelope 012" } ], "@timestamp" : "2024-12-13T14:43:18.985Z", "soap_in" : [ { "log_timestamp" : "2024-12-14 12:00:01", "soap_in" : "Envelope 011" } ] } }, { "_index" : "app-aggregate-2024.12.13", "_type" : "_doc", "_id" : "rop5wJMB81mNBoMqbTwN", "_score" : 1.0, "_source" : { "@version" : "1", "soap_out" : [ { "log_timestamp" : "2024-12-14 12:05:04", "soap_out" : "Envelope 022" } ], "@timestamp" : "2024-12-13T14:43:23.504Z", "soap_in" : [ { "log_timestamp" : "2024-12-14 12:05:03", "soap_in" : "Envelope 021" } ] } } ] } } ``` So in result elasticsearch, we can see that we have 2 values instead of 4, that's great but i still don't know why this never shown the field thread\_id even if it is mentioned in aggregate Thank you in advance, ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [December 13, 2024, 3:49pm UTC](https://discuss.elastic.co/t/logstash-field-is-never-shown-after-aggregation/371963/2 "2024-12-13T15:49:03Z")
> [@Hannah\_J\_Swystun](#): > > ```auto > event.set('soap_in', map['soap_in']) > event.set('soap_out', map['soap_out']) > event.set('thread_id', event.get('thread_id')) > event.cancel() > end > " > push_previous_map_as_event => true > > ``` The three event.set calls have no effect and can be deleted. They add fields to an event that is immediately discarded. Then, when an event with a different task\_id (thread\_id) arrives, a new event is created from the contents of the map, which does not include thread\_id. Try adding `map['thread_id'] = event.get('thread_id')` at the top of the code block. You are not relying on a timeout, so I don't think the timeout\_task\_id\_field option can be used to shortcut this. ---
**Author:** ![Hannah\_J\_Swystun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hannah_j_swystun/32/139547_2.png) [@Hannah\_J\_Swystun](https://discuss.elastic.co/u/Hannah_J_Swystun)\ **Post date:** [December 16, 2024, 8:49am UTC](https://discuss.elastic.co/t/logstash-field-is-never-shown-after-aggregation/371963/3 "2024-12-16T08:49:32Z")
Thank you M.BADGET, it works with timeout =\> 3, when i delete this line, i should wait to see other threads coming! Just one more question, this sample is very short sample with only 4 rows, when it comes to thousands and thousands of log rows ? do this kind of filter could work ? thank you in advance ```auto filter { grok { match => { "message" => [ '%{TIMESTAMP_ISO8601:log_timestamp} thread-%{INT:thread_id} SOAP message <<(?.*?)>>', '%{TIMESTAMP_ISO8601:log_timestamp} thread-%{INT:thread_id} SOAP message >>(?.*?)<<' ] } } aggregate { task_id => "%{thread_id}" code => " map['soap_in'] ||= [] map['soap_out'] ||= [] map['thread_id'] ||= [] map['thread_id'] = event.get('thread_id') if event.get('soap_in') map['soap_in'] << {'soap_in' => event.get('soap_in'), 'log_timestamp' => event.get('log_timestamp')} end if event.get('soap_out') map['soap_out'] << {'soap_out' => event.get('soap_out'), 'log_timestamp' => event.get('log_timestamp')} end if map['soap_in'] && map['soap_out'] event.set('thread_id', map['thread_id']) event.set('soap_in', map['soap_in']) event.set('soap_out', map['soap_out']) event.cancel() end " push_previous_map_as_event => true timeout => 3 } mutate { remove_field => ["message"] } } ```