# Logstash Aggregate map working only for few rows while merging two data sources
**URL:** https://discuss.elastic.co/t/logstash-aggregate-map-working-only-for-few-rows-while-merging-two-data-sources/231293
**Category:** Logstash
**Created:** [May 6, 2020, 8:33am UTC](https://discuss.elastic.co/t/logstash-aggregate-map-working-only-for-few-rows-while-merging-two-data-sources/231293 "2020-05-06T08:33:15Z")
**Posts on this page:** 3
**Page:** 1
### Author:  [@Nani\_20](https://discuss.elastic.co/u/Nani_20)
#### Post date: [May 6, 2020, 8:33am UTC](https://discuss.elastic.co/t/logstash-aggregate-map-working-only-for-few-rows-while-merging-two-data-sources/231293/1 "2020-05-06T08:33:15Z")
I am using the latest version of logstash(7.6.2). I am trying to **merge rows from two different files** using a common ID. Find below the samples of the two data sources. Also, find below my desired output.
**Sample Data 1** with columns **ID, Country, State**
```auto
111 US NY
112 IN KA
113 US MA
```
**Sample Data 2** with columns **ID and Info**
```auto
111 abc
111 abd
112 xyz
112 xya
113 qwe
113 qwr
```
**Desired Output with merge**
```auto
111 abc US NY
111 abd US NY
112 xyz IN KA
112 xya IN KA
113 qwe US MA
113 qwr US MA
```
I have tried using aggregate as shown in the below config file. I have also set pipeline workers to 1 and disabled java execution by adding the following commands in logstash.yml
```auto
pipeline.workers: 1
pipeline.java_execution: false
```
**Config File**
```auto
if [Country] =~ /.+/ {
aggregate {
task_id => "%{ID}"
code => "
map['country'] = event.get('Country')
map['state'] = event.get('State')
event.cancel()
"
}
drop{}
}
if ![Country] {
aggregate {
task_id => "%{ID}"
code => "
event.set('Country', map['country'])
event.set('State', map['state'])
"
}
}
```
I was able to obtain the merge results for the sample but when i try it on entire data only few rows get updated with the merged columns instead of all the rows.
I am I missing anything here? Any help here is appreciated. TIA
---
### Author:  [@Nani\_20](https://discuss.elastic.co/u/Nani_20)
#### Post date: [May 7, 2020, 9:02am UTC](https://discuss.elastic.co/t/logstash-aggregate-map-working-only-for-few-rows-while-merging-two-data-sources/231293/2 "2020-05-07T09:02:55Z")
In case if anyone is looking for this. I found a workaround!
Used **Elasticsearch Filter Plugin** as below to merge the two indexes.
**Config File**
```auto
elasticsearch {
hosts => ["http://localhost:9200"]
index => "mapping_file"
query => "ID:%{[ID]}"
fields => {"COUNTRY" => "COUNTRY"}
}
```
Pushed the data 1 into ElasticSearch as **mapping\_file** and used it in the **ElasticSearch Filter plugin** while pushing data 2
Hope it helps!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 4, 2020, 9:03am UTC](https://discuss.elastic.co/t/logstash-aggregate-map-working-only-for-few-rows-while-merging-two-data-sources/231293/3 "2020-06-04T09:03:08Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.