# Aggregate filter plugin events not adding to elasticsearch index

**URL:** <https://discuss.elastic.co/t/aggregate-filter-plugin-events-not-adding-to-elasticsearch-index/189610>\
**Category:** Logstash\
**Created:** [July 9, 2019, 7:01pm UTC](https://discuss.elastic.co/t/aggregate-filter-plugin-events-not-adding-to-elasticsearch-index/189610 "2019-07-09T19:01:59Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![rohith1570](https://avatars.discourse-cdn.com/v4/letter/r/3ab097/32.png) [@rohith1570](https://discuss.elastic.co/u/rohith1570)\
**Post date:** [July 9, 2019, 7:01pm UTC](https://discuss.elastic.co/t/aggregate-filter-plugin-events-not-adding-to-elasticsearch-index/189610/1 "2019-07-09T19:01:59Z")

</div>

Hi,

In stdout i'm able to see the Aggregate events got printed, but not able to see in elasticsearch index.

logstash conf:

```
filter {
    if [user.id] != "" and [user.id] != 'null' and [user.id] != 'error' {
    aggregate {
    task_id => "%{user.id}_%{+d}_%{+MMM}"
    code => "
    if(event.get('user.id') != 'error')
    map['sdfilesize'] ||= 0 ;
    fileSize = event.get('file.size')
    if fileSize
      map['sdfilesize'] += event.get('file.size');
    end
    map['user.id'] ||= 0 ;
    map['user.id'] = event.get('user.id');
    map['incident_creation_date'] ||= 0 ; map['incident_creation_date'] = event.get('incident_creation_date');
    map['user.business_unit'] ||= 0 ; map['user.business_unit'] = event.get('user.business_unit');
    map['user.full_name'] ||= 0 ; map['user.full_name'] = event.get('user.full_name');
    map['recipient_identifier'] ||= [] ; map['recipient_identifier'] << event.get('recipient_identifier');
    map['file.extension'] ||= [] ; map['file.extension'] << event.get('file.extension');
    map['user.department'] ||= 0 ; map['user.department'] = event.get('user.department');
    end"
    push_map_as_event_on_timeout => true
    timeout_task_id_field => "sdf"
    timeout => 300
    inactivity_timeout => 120
    timeout_tags => ['_aggregatetimeout']
      }
     }
 }

output {
      stdout { codec => rubydebug }
 
    elasticsearch {
      ssl => true
      ssl_certificate_verification => false
      hosts => "<url>"
      user => "logstash"
      password => "logstash"
      index => "testidx-%{+YYYY.MM}"
      document_id => "%{id}"
      routing => "%{user.id}"
    }

  }
```

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [July 9, 2019, 7:08pm UTC](https://discuss.elastic.co/t/aggregate-filter-plugin-events-not-adding-to-elasticsearch-index/189610/2 "2019-07-09T19:08:00Z")

</div>

> [@rohith1570](#):
>
> document\_id =\> "%{id}"

Where are you setting the `id` field? If you do not you will likely end up with a single document being repeatedly updated.

---

<div class="post-metadata">

**Author:** ![rohith1570](https://avatars.discourse-cdn.com/v4/letter/r/3ab097/32.png) [@rohith1570](https://discuss.elastic.co/u/rohith1570)\
**Post date:** [July 9, 2019, 7:12pm UTC](https://discuss.elastic.co/t/aggregate-filter-plugin-events-not-adding-to-elasticsearch-index/189610/3 "2019-07-09T19:12:56Z")

</div>

Setting it inside filter conf.  
Sorry, forgot the conf to add.

```
    fingerprint {
      concatenate_sources => true
      key => "logs"
      source => ["user.first_name", "user.last_name", "user.email", "log_id", "file.name"]
      target => "id"
      method => "SHA256"
    }
```

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [July 9, 2019, 7:18pm UTC](https://discuss.elastic.co/t/aggregate-filter-plugin-events-not-adding-to-elasticsearch-index/189610/4 "2019-07-09T19:18:03Z")

</div>

I do not see how events created by the aggregate filter would have these fields. How can the `id` field be calculated for these events? Is it not always going to end up being the same value? If you look at the statistics for your index, do you see a lot of updates/deletes?

---

<div class="post-metadata">

**Author:** ![rohith1570](https://avatars.discourse-cdn.com/v4/letter/r/3ab097/32.png) [@rohith1570](https://discuss.elastic.co/u/rohith1570)\
**Post date:** [July 9, 2019, 7:42pm UTC](https://discuss.elastic.co/t/aggregate-filter-plugin-events-not-adding-to-elasticsearch-index/189610/5 "2019-07-09T19:42:16Z")

</div>

> [@Christian\_Dahlqvist](#):
>
> How can the `id` field be calculated for these events?

For the `task id` i'm taking `user.id` not `id`.

> [@Christian\_Dahlqvist](#):
>
> Is it not always going to end up being the same value?

No, it wont be same, because in the finger print it also has `log_id`, which is unique.  
And i also tried removing `document_id` from output conf, still aggregate events not getting added.

> [@Christian\_Dahlqvist](#):
>
> If you look at the statistics for your index, do you see a lot of updates/deletes?

I don't see any deletions, attached stats please let me know if you find anything .

> Blockquote  
> ` "primaries": { "docs": { "count": 32, "deleted": 0 }, "store": { "size_in_bytes": 110226 }, "indexing": { "index_total": 32, "index_time_in_millis": 27, "index_current": 0, "index_failed": 0, "delete_total": 0, "delete_time_in_millis": 0, "delete_current": 0, "noop_update_total": 0, "is_throttled": false, "throttle_time_in_millis": 0 }, "get": { "total": 0, "time_in_millis": 0, "exists_total": 0, "exists_time_in_millis": 0, "missing_total": 0, "missing_time_in_millis": 0, "current": 0 }, "search": { "open_contexts": 0, "query_total": 9, "query_time_in_millis": 4, "query_current": 0, "fetch_total": 2, "fetch_time_in_millis": 22, "fetch_current": 0, "scroll_total": 0, "scroll_time_in_millis": 0, "scroll_current": 0, "suggest_total": 0, "suggest_time_in_millis": 0, "suggest_current": 0 }, "merges": { "current": 0, "current_docs": 0, "current_size_in_bytes": 0, "total": 0, "total_time_in_millis": 0, "total_docs": 0, "total_size_in_bytes": 0, "total_stopped_time_in_millis": 0, "total_throttled_time_in_millis": 0, "total_auto_throttle_in_bytes": 41943040 }, "refresh": { "total": 17, "total_time_in_millis": 59, "listeners": 0 }, "flush": { "total": 3, "periodic": 0, "total_time_in_millis": 49 }, "warmer": { "current": 0, "total": 9, "total_time_in_millis": 0 }, "query_cache": { "memory_size_in_bytes": 0, "total_count": 0, "hit_count": 0, "miss_count": 0, "cache_size": 0, "cache_count": 0, "evictions": 0 }, "fielddata": { "memory_size_in_bytes": 0, "evictions": 0 }, "completion": { "size_in_bytes": 0 }, "segments": { "count": 2, "memory_in_bytes": 54638, "terms_memory_in_bytes": 45578, "stored_fields_memory_in_bytes": 624, "term_vectors_memory_in_bytes": 0, "norms_memory_in_bytes": 7168, "points_memory_in_bytes": 44, "doc_values_memory_in_bytes": 1224, "index_writer_memory_in_bytes": 0, "version_map_memory_in_bytes": 0, "fixed_bit_set_memory_in_bytes": 0, "max_unsafe_auto_id_timestamp": -1, "file_sizes": {} }, "translog": { "operations": 32, "size_in_bytes": 67036, "uncommitted_operations": 0, "uncommitted_size_in_bytes": 110, "earliest_last_modified_age": 0 }, "request_cache": { "memory_size_in_bytes": 770, "evictions": 0, "hit_count": 3, "miss_count": 2 }`

Hope this helps, please let me know.

---

<div class="post-metadata">

**Author:** ![rohith1570](https://avatars.discourse-cdn.com/v4/letter/r/3ab097/32.png) [@rohith1570](https://discuss.elastic.co/u/rohith1570)\
**Post date:** [July 10, 2019, 1:30pm UTC](https://discuss.elastic.co/t/aggregate-filter-plugin-events-not-adding-to-elasticsearch-index/189610/6 "2019-07-10T13:30:22Z")

</div>

@Christian_Dahlqvist  
Hi Chris,

If you can go through my previous replies, that would be great!  
Please do let me know, if you need any more info to debug.

---

<div class="post-metadata">

**Author:** ![rohith1570](https://avatars.discourse-cdn.com/v4/letter/r/3ab097/32.png) [@rohith1570](https://discuss.elastic.co/u/rohith1570)\
**Post date:** [July 10, 2019, 6:11pm UTC](https://discuss.elastic.co/t/aggregate-filter-plugin-events-not-adding-to-elasticsearch-index/189610/7 "2019-07-10T18:11:53Z")

</div>

@Badger Can you please help me with this.?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 10, 2019, 6:34pm UTC](https://discuss.elastic.co/t/aggregate-filter-plugin-events-not-adding-to-elasticsearch-index/189610/8 "2019-07-10T18:34:27Z")

</div>

As Christian pointed out, none of the fields you are using to generate the fingerprint are part of the event.

The aggregate filter is generating the event when there is a timeout. The event will have the fields that you added to the map. That is,

user.id, incident\_creation\_date, user.business\_unit, user.full\_name,  
recipient\_identifier, file.extension, user.department, sdf

It will not have any of the fields you are using for the fingerprint.

source =\> ["user.first\_name", "user.last\_name", "user.email", "log\_id", "file.name"]

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [July 10, 2019, 6:38pm UTC](https://discuss.elastic.co/t/aggregate-filter-plugin-events-not-adding-to-elasticsearch-index/189610/9 "2019-07-10T18:38:26Z")

</div>

An easy way to test whether it is the fingerprint filter causing problems is to simply comment it out.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 7, 2019, 6:38pm UTC](https://discuss.elastic.co/t/aggregate-filter-plugin-events-not-adding-to-elasticsearch-index/189610/10 "2019-08-07T18:38:28Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.