# Fingerprint & Aggregate Filter

**URL:** <https://discuss.elastic.co/t/fingerprint-aggregate-filter/125979>\
**Category:** Logstash\
**Created:** [March 28, 2018, 6:02pm UTC](https://discuss.elastic.co/t/fingerprint-aggregate-filter/125979 "2018-03-28T18:02:28Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![apatel2848](https://avatars.discourse-cdn.com/v4/letter/a/a88e4f/32.png) [@apatel2848](https://discuss.elastic.co/u/apatel2848)\
**Post date:** [March 28, 2018, 6:02pm UTC](https://discuss.elastic.co/t/fingerprint-aggregate-filter/125979/1 "2018-03-28T18:02:28Z")

</div>

How to generate a unique fingerprint for every record retrieved via JDBC plugin? For example, I am retrieving list of invoice line items via a JDBC plugin and using an aggregate filter to group those line items by the invoice number. Before I index the the document in elasticsearch, I would like to assign a unique fingerprint id as the document\_id.

The issue is, if I use a fingerprint filter before the aggregate filter than the generated fingerprint value is not available either in event object or in @metadata object within the output section. If I use a fingerprint filter after the aggregate than the same fingerprint value is getting assigned to all document\_id fields.

Any idea??

---

<div class="post-metadata">

**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\
**Post date:** [March 29, 2018, 9:16pm UTC](https://discuss.elastic.co/t/fingerprint-aggregate-filter/125979/2 "2018-03-29T21:16:43Z")

</div>

It may be helpful to share your pipeline configuration.

> [@apatel2848](#):
>
> unique fingerprint

If the same record is retrieved twice, do you expect the same fingerprint, or a different one?

> [@apatel2848](#):
>
> if I use a fingerprint filter before the aggregate filter than the generated fingerprint value is not available either in event object

The aggregate filter is a little magical, in that it "swallows" many events, correlates them on the given task id, using your code to set and manipulate values in a map for that particular task, and then eventually emits _different_ events once a "task" is deemed complete (that is, when it sees a "final" event or reaches a timeout).

As such, it requires that you copy over the bits that you want and put them in the map; while many individual events go into the aggregate filters, only one event will come out per ending event (or timeout).

> [@apatel2848](#):
>
> If I use a fingerprint filter after the aggregate than the same fingerprint value is getting assigned to all document\_id fields.

Again, I would need to see how your fingerprint is defined; if the event emitted by the closing event in the aggregate filter does not have a particular key set, we would be effectively be doing a consistent hashing of a consistent (empty) value, resulting in a single consistent key.

---

<div class="post-metadata">

**Author:** ![apatel2848](https://avatars.discourse-cdn.com/v4/letter/a/a88e4f/32.png) [@apatel2848](https://discuss.elastic.co/u/apatel2848)\
**Post date:** [March 29, 2018, 9:34pm UTC](https://discuss.elastic.co/t/fingerprint-aggregate-filter/125979/3 "2018-03-29T21:34:31Z")

</div>

> [@yaauie](#):
>
> twice

I meant unique for every invoice.

---

<div class="post-metadata">

**Author:** ![apatel2848](https://avatars.discourse-cdn.com/v4/letter/a/a88e4f/32.png) [@apatel2848](https://discuss.elastic.co/u/apatel2848)\
**Post date:** [March 29, 2018, 9:39pm UTC](https://discuss.elastic.co/t/fingerprint-aggregate-filter/125979/4 "2018-03-29T21:39:41Z")

</div>

Here is the conf file. I want to generate a fingerprint for each invoice document based on its PK and assign the same fingerprint value to the document\_id

Assume 'invoicePK' is one of the returned column.

```
input {
  jdbc {
    jdbc_connection_string => "jdbc:sqlserver://........"
    # The user we wish to execute our statement as
    jdbc_user => "xxxxxx"
    jdbc_password => "xxxxxxxxx"
    # The path to our downloaded jdbc driver
    jdbc_driver_library => "xxxxxxxx.jar"
    jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
    # our query
    statement => "xxxxxxxxxxxxx"
  }
}
filter {

  fingerprint {
    "method" => "SHA1"
    "key" => "dummyPhrase"
    "source" => ['%{invoicePK}']
  }

  aggregate {
    task_id => "%{invoicePK}"
    code =>
    "
      map['id'] = event.get('fingerprint')
      map['pk'] = event.get('invoicePK')
      map['number'] = event.get('number')
      map['date'] = event.get('date')
      map['dueDate'] = event.get('duedate')
      map['total'] = event.get('total')
      map['subTotal'] = event.get('subtotal')
      map['tax'] = event.get('tax')
      map['freight'] = event.get('freight')
      map['discount'] = event.get('discount')
      map['createdOn'] = event.get('createdon')
      map['modifiedOn'] = event.get('modifiedon')

      map['vendor'] ||= {}
      map['vendor']['id'] = event.get('vendorid');
      map['vendor']['name'] = event.get('vendor')

      map['lineItems'] ||= []
      map['lineItems'] << {
        'productId' => event.get('productid'),
        'vendorProductCode' => event.get('vendorproductcode'),
        'category' => event.get('category'),
        'description' => event.get('description'),
        'qty' => event.get('qty'),
        'unitCost' => event.get('unitcost'),
        'total' => event.get('extendedtotal')
      }

      event.cancel()
    "
    push_previous_map_as_event => true
    timeout => 3
  }

  mutate {
    add_tag => ["logstash 6.2.3"]
    remove_tag => ["_aggregatefinalflush"]
  }
}
output {
  stdout { codec => rubydebug { metadata => true } }

  elasticsearch {
    hosts => "https://localhost"
    index => "invoice"
  }
}
```

---

<div class="post-metadata">

**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\
**Post date:** [March 31, 2018, 12:26am UTC](https://discuss.elastic.co/t/fingerprint-aggregate-filter/125979/5 "2018-03-31T00:26:14Z")

</div>

What about the provided configuration isn't working? What output do you expect to see, and how does what you get differ? When I run similar, I see a bunch of events with `id`s that are copied from the generated fingerprints.

A couple notes from my brief dive into the `aggregate` filter:

- The `push_previous_map_as_event` setting requires that the events coming back are very explicitly ordered -- that is, that once you get a row that has a different value for `invoicePK`, that you'll never get any more rows with that value.
- If you've got more than one pipeline worker, or running without explicit ordering constrained in Logstash, you're going to have a bad time; this plugin is very tightly bound to sequence and ordering.

---

<div class="post-metadata">

**Author:** ![apatel2848](https://avatars.discourse-cdn.com/v4/letter/a/a88e4f/32.png) [@apatel2848](https://discuss.elastic.co/u/apatel2848)\
**Post date:** [March 31, 2018, 4:32pm UTC](https://discuss.elastic.co/t/fingerprint-aggregate-filter/125979/6 "2018-03-31T16:32:12Z")

</div>

With the above config, I get the same fingerprint for multiple invoices. When I look at the map attributes, 'fingerprint' key has the same value for all documents but 'pk' key has different values. I expect to see different fingerprint values for each invoices. I forgot to mention a line in an output section. I do have document\_id = "%{fingerprint}" for the elasticsearch output. Only one document is getting indexed because of the same fingerprint value for every document.

---

<div class="post-metadata">

**Author:** ![apatel2848](https://avatars.discourse-cdn.com/v4/letter/a/a88e4f/32.png) [@apatel2848](https://discuss.elastic.co/u/apatel2848)\
**Post date:** [March 31, 2018, 5:47pm UTC](https://discuss.elastic.co/t/fingerprint-aggregate-filter/125979/7 "2018-03-31T17:47:28Z")

</div>

All my events are in order and will make sure it remains that way. Thanks for clarifying a behavior of the aggregate filter.

---

<div class="post-metadata">

**Author:** ![apatel2848](https://avatars.discourse-cdn.com/v4/letter/a/a88e4f/32.png) [@apatel2848](https://discuss.elastic.co/u/apatel2848)\
**Post date:** [April 4, 2018, 7:05pm UTC](https://discuss.elastic.co/t/fingerprint-aggregate-filter/125979/8 "2018-04-04T19:05:59Z")

</div>

When you run, does each document has a unique fingerprint? In my case, I get the same fingerprint for all documents as it seems that fingerprint routine only gets called once only.

---

<div class="post-metadata">

**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\
**Post date:** [April 4, 2018, 9:19pm UTC](https://discuss.elastic.co/t/fingerprint-aggregate-filter/125979/9 "2018-04-04T21:19:59Z")

</div>

AHA!

The [`source` directive of the `fingerprint` plugin](https://www.elastic.co/guide/en/logstash/current/plugins-filters-fingerprint.html#plugins-filters-fingerprint-source) takes a literal field reference, not a sprintf format string:

```diff
   fingerprint {
     "method" => "SHA1"
     "key" => "dummyPhrase"
- "source" => ['%{invoicePK}']
+ "source" => ['invoicePK']
   }

```

---

<div class="post-metadata">

**Author:** ![apatel2848](https://avatars.discourse-cdn.com/v4/letter/a/a88e4f/32.png) [@apatel2848](https://discuss.elastic.co/u/apatel2848)\
**Post date:** [April 4, 2018, 10:18pm UTC](https://discuss.elastic.co/t/fingerprint-aggregate-filter/125979/10 "2018-04-04T22:18:44Z")

</div>

Thanks, finally I get a distinct fingerprint for each invoice. But now I see a different problem. Despite of having an order by INVPK clause in my query, aggregate is not aggregating documents by invoicePK.  
I have two invoices with with three line items each. My query is returning list of line items ordered by invoicePK. I am expecting two invoice documents will be formed with three line items in an lineItem array. But instead I have four invoice documents. First invoice is with just one line item for PK1. Second invoice is with just one line item for PK2. Third invoice is with two line items for PK1 and fourth invoice document is with two line items for PK2.

Why do you think aggregate is not working despite of having events in order by invoicePK?

---

<div class="post-metadata">

**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\
**Post date:** [April 5, 2018, 12:53am UTC](https://discuss.elastic.co/t/fingerprint-aggregate-filter/125979/11 "2018-04-05T00:53:48Z")

</div>

Are you starting Logstash with just one worker thread? The aggregate filter is order-dependent, and order cannot be guaranteed when multiple worker threads are splitting the work.

---

<div class="post-metadata">

**Author:** ![apatel2848](https://avatars.discourse-cdn.com/v4/letter/a/a88e4f/32.png) [@apatel2848](https://discuss.elastic.co/u/apatel2848)\
**Post date:** [April 5, 2018, 1:06am UTC](https://discuss.elastic.co/t/fingerprint-aggregate-filter/125979/12 "2018-04-05T01:06:02Z")

</div>

I had default 2 worker threads running. After changing it to 1 thread, it is now working as expected. Does that mean, an aggregate filter can't take the advantage of multi-threading?

---

<div class="post-metadata">

**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\
**Post date:** [April 5, 2018, 1:17am UTC](https://discuss.elastic.co/t/fingerprint-aggregate-filter/125979/13 "2018-04-05T01:17:14Z")

</div>

Yes; the aggregate filter clearly calls out in the docs that it does not support parallel execution.

> You should be very careful to set Logstash filter workers to 1 (`-w 1` flag) for this filter to work correctly otherwise events may be processed out of sequence and unexpected results will occur.
> 
> -- [Logstash Aggregate Filter docs](https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html#plugins-filters-aggregate-description)

---

<div class="post-metadata">

**Author:** ![apatel2848](https://avatars.discourse-cdn.com/v4/letter/a/a88e4f/32.png) [@apatel2848](https://discuss.elastic.co/u/apatel2848)\
**Post date:** [April 5, 2018, 2:21am UTC](https://discuss.elastic.co/t/fingerprint-aggregate-filter/125979/14 "2018-04-05T02:21:32Z")

</div>

Thanks for the info and all the support

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 3, 2018, 2:21am UTC](https://discuss.elastic.co/t/fingerprint-aggregate-filter/125979/15 "2018-05-03T02:21:32Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.