# Aggregate filter logstash- pulling field from another event

**URL:** <https://discuss.elastic.co/t/aggregate-filter-logstash-pulling-field-from-another-event/146850>\
**Category:** Logstash\
**Created:** [August 31, 2018, 10:56am UTC](https://discuss.elastic.co/t/aggregate-filter-logstash-pulling-field-from-another-event/146850 "2018-08-31T10:56:00Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![thiyagarajan.vasudev](https://avatars.discourse-cdn.com/v4/letter/t/e9a140/32.png) [@thiyagarajan.vasudev](https://discuss.elastic.co/u/thiyagarajan.vasudev)\
**Post date:** [August 31, 2018, 10:56am UTC](https://discuss.elastic.co/t/aggregate-filter-logstash-pulling-field-from-another-event/146850/1 "2018-08-31T10:56:00Z")

</div>

I am sending logs from filebeat to logstash and in logstash I am doing filter based on regex and storing it in elastic search  
My logstash config:  
input {  
beats {  
port =\> 5002  
}  
}

filter  
{  
grok {  
add\_tag =\> ["first"]  
match =\> { "message" =\> ""._.definitionName": "(?\<build\_DefinitionName\>._?)"" }  
}  
grok {  
add\_tag =\> ["first"]  
match =\> { "message" =\> ""._.requestedFor": "(?\<build\_RequesterName\>._?)"" }  
}  
grok {  
add\_tag =\> ["first"]  
match =\> { "message" =\> ""._.buildNumber": "(?\<build\_BuildNumber\>._?)"" }  
}  
grok {  
add\_tag =\> ["first"]  
match =\> { "message" =\> ""._.teamProject": "(?\<build\_TeamProject\>._?)"" }  
}  
grok {  
add\_tag =\> ["first"]  
match =\> { "message" =\> "(?\<build\_ErrorMessage\> ERR .\*)" }  
}  
grok {  
add\_tag =\> ["end"]  
match =\> { "message" =\> "Current state: job state = '(?\<build\_IsFailed\>.+)'" }  
}  
if "end" in [tags]{  
aggregate {  
task\_id =\> "%{source}"  
code =\> "event.set('build\_DefinitionName', '%{build\_DefinitionName}')  
event.set('key', 'value')  
map['build\_DefinitionName'] ||= event.get('build\_DefinitionName')  
event.set('build\_BuildNumber', map['build\_BuildNumber'])  
event.set('build\_RequesterName', map['build\_RequesterName'])  
event.set('build\_TeamProject', map['build\_TeamProject'])  
event.set('build\_IsFailed', map['build\_IsFailed'])  
event.set('build\_ErrorMessage', map['build\_ErrorMessage'])"  
map\_action =\> "create\_or\_update"  
push\_previous\_map\_as\_event =\> true  
end\_of\_task =\> false  
}  
}  
if "first" not in [tags] and "end" not in [tags] {  
drop { }  
}  
mutate {  
remove\_field =\> ["message"]  
}  
mutate {  
remove\_tag =\> ["first"]  
}  
}

output {  
elasticsearch {  
hosts =\> ["elasticsearch:9200"]  
index =\> "mylog-%{+YYYY.MM.dd}"  
}  
stdout { codec =\> rubydebug }  
}  
Since we are sending data line by line from filebeat. Each of the above fields been stored as separate event so at the end we are trying to aggregate but there is no value against the fields.

For example:  
build\_DefinitionName filed available in the last event but no value:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/8/0851b486724e891cd979a701e909b0c67a5e4942.png)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 28, 2018, 10:56am UTC](https://discuss.elastic.co/t/aggregate-filter-logstash-pulling-field-from-another-event/146850/2 "2018-09-28T10:56:03Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.