# Aggregate filter to remember field accross lines
**URL:** https://discuss.elastic.co/t/aggregate-filter-to-remember-field-accross-lines/184712
**Category:** Logstash
**Created:** [June 7, 2019, 8:43am UTC](https://discuss.elastic.co/t/aggregate-filter-to-remember-field-accross-lines/184712 "2019-06-07T08:43:29Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@msk\_76](https://discuss.elastic.co/u/msk_76)
#### Post date: [June 7, 2019, 8:43am UTC](https://discuss.elastic.co/t/aggregate-filter-to-remember-field-accross-lines/184712/1 "2019-06-07T08:43:29Z")
I have to remember date from one line and use it while processing event\_timestamp filed on other lines of log file. For this I am using aggregate filter. Please guide what I am doing wrong here or any other way to do it?
My desired output is event\_timestamp should contain where is the remembered value from TIMESTAMP line and is the event time of message in the log file.
**My data :**
14:03:03 (lmgrd) TIMESTAMP 5/13/2019
15:56:24 (cdslmd) OUT: "PKC0603" mnarasim@dlhsx00012
15:56:32 (cdslmd) OUT: "Analog\_Design\_Environment\_XL" gautamd@dlhsx00005
**My config :**
filter {
grok {
match =\> ["message", "%{DATA:event\_timestamp} (%{DATA:lmgrd}) TIMESTAMP %{DATE:monthday}"]
tag\_on\_failure =\> ["message\_data"]
}
```
mutate {
add_field => { "taskId" => "all" }
}
if "message_data" not in [tags] {
aggregate {
task_id => "%{taskId}"
code => "map['monthday'] = event.get('monthday')"
}
}
else {
aggregate {
task_id => "%{taskId}"
code => "try = event.set('monthday', map['monthday'])"
map_action => "update"
}
grok {
match => ["message", "%{TIME:event_timestamp} \(%{DATA:lic_vendor_name}\) (?(OUT|IN|DENIED))\: \"%{DATA:lic_feature_name}\" %{DATA:user_name}@%{HOSTNAME:host_name}"] }
mutate { replace => ["event_timestamp", "%{event_timestamp} %{try}"] }
}
```
**My output**
"event\_timestamp" =\> "15:56:24 %{try}"
```
"user_name" => "gautamd",
"source" => "/home/msk/cadence-cdslmd-dlhl0939_dlhl0940_dlhl0941.log",
"message" => "15:56:32 (cdslmd) OUT: \"Analog_Design_Environment_XL\" gautamd@dlhsx00005 ",
"beat" => {
"name" => "dlhl2117",
"hostname" => "dlhl2117",
"version" => "6.4.2"
},
"@timestamp" => 2019-06-07T08:16:28.915Z,
"host" => {
"name" => "dlhl2117"
},
"fields" => {
"document_type" => "fle-type"
},
"taskId" => "all",
"lic_feature_name" => "Analog_Design_Environment_XL",
"offset" => 92,
"event_timestamp" => "15:56:32 %{try}"
```
}
{
"event\_type" =\> "OUT",
"@version" =\> "1",
"host\_name" =\> "dlhsx00012",
"prospector" =\> {
"type" =\> "log"
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [June 7, 2019, 1:25pm UTC](https://discuss.elastic.co/t/aggregate-filter-to-remember-field-accross-lines/184712/2 "2019-06-07T13:25:11Z")
You are assigning the return value of event.set to a variable called try, which is only accessible in the code block of the ruby filter. I think you want something more like
```
grok {
match => ["message", "%{DATA:event_timestamp} \(%{DATA:lmgrd}\) TIMESTAMP %{DATE:monthday}"]
tag_on_failure => ["message_data"]
}
mutate {
add_field => { "taskId" => "all" }
}
if "message_data" not in [tags] {
aggregate {
task_id => "%{taskId}"
code => "map['monthday'] = event.get('monthday')"
}
} else {
aggregate {
task_id => "%{taskId}"
code => "event.set('monthday', map['monthday'])"
map_action => "update"
}
grok { match => ["message", "%{TIME:event_timestamp} \(%{DATA:lic_vendor_name}\) (?(OUT|IN|DENIED))\: \"%{DATA:lic_feature_name}\" %{DATA:user_name}@%{HOSTNAME:host_name}"] }
}
mutate { replace => ["event_timestamp", "%{event_timestamp} %{monthday}"] }
```
---
### Author:  [@msk\_76](https://discuss.elastic.co/u/msk_76)
#### Post date: [June 10, 2019, 10:57am UTC](https://discuss.elastic.co/t/aggregate-filter-to-remember-field-accross-lines/184712/3 "2019-06-10T10:57:54Z")
I modified the logstash config slightly but still the aggregate filter is unable to remember the value of map['monthday']. Here is my update :
My data which is in filename "cadence-cdslmd-dlhl0939\_dlhl0940\_dlhl0941-5.11.2019-event.log"
15:56:24 (cdslmd) OUT: "PKC0603" mnarasim@dlhsx00012
15:56:32 (cdslmd) OUT: "Analog\_Design\_Environment\_XL" gautamd@dlhsx00005
15:56:33 (lmgrd) TIMESTAMP 5/13/2019
15:56:35 (cdslmd) OUT: "OASIS\_Simulation\_Interface" mnarasim@dlhsx00012
15:56:37 (cdslmd) IN: "111" guptasri@dlhsx00010
15:56:42 (cdslmd) OUT: "PKC0603" mnarasim@dlhsx00012
My config
filter {
```
grok {
match => ["message", "%{DATA:event_timestamp} \(%{DATA:lmgrd}\) TIMESTAMP %{DATE:monthday}"]
tag_on_failure => ["message_data"]
}
mutate {
add_field => { "taskId" => "all" }
}
if "message_data" not in [tags] {
aggregate {
task_id => "%{taskId}"
code => "map['monthday'] = event.get('monthday')"
}
}
else {
aggregate {
task_id => "%{taskId}"
code => "if (map['monthday'] != nil) then event.set('monthday', map['monthday']) else event.set('monthday', (event.get('source').split('-')[3])) end"
}
grok {
match => ["message", "%{TIME:event_timestamp} \(%{DATA:lic_vendor_name}\) (?(OUT|IN|DENIED))\: \"%{DATA:lic_feature_name}\" %{DATA:user_name}@%{HOSTNAME:host_name}"] }
mutate { replace => ["event_timestamp", "%{monthday} %{event_timestamp}"] }
date { match => ["event_timestamp", "M/d/yyyy HH:mm:ss","M.d.yyyy HH:mm:ss","ISO8601"] target => "event_timestamp" }
}
```
}
output {
stdout {
codec =\> rubydebug
}
}
In the output corresponding to input line 5 and line 6
15:56:37 (cdslmd) IN: "111" guptasri@dlhsx00010
15:56:42 (cdslmd) OUT: "PKC0603" mnarasim@dlhsx00012
The aggregate filter is not able to remember the map['monthday'] of previous lines i.e. "5/13/2019".
Output corresponding to input line 5 "event\_timestamp" =\> 2019-05-11T10:26:37.000Z,
Output corresponding to input line 6 "event\_timestamp" =\> 2019-05-11T10:26:42.000Z,
The expected output of
Line 5 should be : "event\_timestamp" =\> 2019-05-13T10:26:37.000Z,
Line 6 should be : ""event\_timestamp" =\> 2019-05-13T10:26:42.000Z,
Am I missing something?
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 8, 2019, 10:57am UTC](https://discuss.elastic.co/t/aggregate-filter-to-remember-field-accross-lines/184712/4 "2019-07-08T10:57:59Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.