# Logstash not aggregating as it should (I think)

**URL:** <https://discuss.elastic.co/t/logstash-not-aggregating-as-it-should-i-think/113509>\
**Category:** Logstash\
**Created:** [December 28, 2017, 9:21pm UTC](https://discuss.elastic.co/t/logstash-not-aggregating-as-it-should-i-think/113509 "2017-12-28T21:21:29Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![ppalmeida](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppalmeida/32/26066_2.png) [@ppalmeida](https://discuss.elastic.co/u/ppalmeida)\
**Post date:** [December 28, 2017, 9:21pm UTC](https://discuss.elastic.co/t/logstash-not-aggregating-as-it-should-i-think/113509/1 "2017-12-28T21:21:30Z")

</div>

Hey, there.

I am trying to use LogStash to send data do ES and I have encountered a problem I could not handle yet:

Consider a table like where the important columns are:

proc\_id | title | operations\_id | products\_id | keywords\_name | keywords\_id

Sample data:

> **[Logstash data sample](https://docs.google.com/spreadsheets/d/1O08N9BooMUbgtpVSDFg1_KgeI9dcttwevAJYRiLwmxE/edit?usp=sharing)**
>
> Página1
> 
> id( as proc\_id), title, operations\_id, products\_id, keywords\_name, keywords\_id
> 510, Procedure 1, 2, 10, name A, 1168
> 510, Procedure 1, 2, 20, name A, 1168
> 510, Procedure 1, 2, 30, name A, 1168
> 510, Procedure 1, 3, 40, name A, 1168
> 510,...

This table, is generated by a query and converted to JSON documents with LogStash + JDBC. The problem is:

When I run logstash, it does not aggregate all the "operations\_id" and "products\_id". It looks like it is skipping some rows. So, considering the table above (in google docs), sometimes the document is stored in ES like this:

```
{
    "id": 510,
    "products_id": [10, 20, 30, 40], 
    etc..
}

```

If I drop the index and run it again, it looks like:

```
 {
     "id": 510,
     "products_id": [50],
     etc..
 }

```

But the correct would be:

```
{
    "id": 510,
    "products_id": [10, 20, 30, 40, 50],
    etc..
}

```

My logstash.conf file looks like this:

```
filter {
    aggregate {
    task_id => "%{proc_id}" # this is the ID of the procedure, the main unique ID.
    code => "
    map['id'] = event.get('proc_id')
    map['title'] = event.get('title')
    map['description_raw'] = event.get('description_raw')
    map['scripts'] ||= []
    map['scripts'] << {
            'title_raw' => event.get('scripts_title_raw'),
            'content_raw' => event.get('scripts_content_raw')
    }
    map['categories'] ||= []
    map['categories'] << event.get('categories_name')
    map['keywords'] ||= []
    map['keywords'] << event.get('keywords_name')
    map['operations'] ||= []
    map['operations'] << event.get('operations_id')
    map['products'] ||= []
    map['products'] << event.get('products_id')
    "
    push_previous_map_as_event => true
    timeout => 5000
    timeout_tags => ['aggregated']
    }

    if "aggregated" not in [tags] {
            drop {}
    }

    # Remove fields that are not needed in ElasticSearch:
    else {
            mutate {
                    remove_field => ["proc_id", "categories_name", "categories_id", "keywords_id", "keywords_name", "scripts_id", "operations_id", "products_id"]
    }
    }
    }

```

Any help is very appreciated. I do not know how to effectively aggregate all the "products\_id" and "operations\_id", "keyword\_names" in one single document into ES.

Thank you.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 25, 2018, 9:21pm UTC](https://discuss.elastic.co/t/logstash-not-aggregating-as-it-should-i-think/113509/2 "2018-01-25T21:21:38Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.