# Using my own document\_id - is there a faster way?

**URL:** <https://discuss.elastic.co/t/using-my-own-document-id-is-there-a-faster-way/108667>\
**Category:** Logstash\
**Created:** [November 22, 2017, 8:12am UTC](https://discuss.elastic.co/t/using-my-own-document-id-is-there-a-faster-way/108667 "2017-11-22T08:12:16Z")\
**Posts on this page:** 7\
**Page:** 2

<div class="post-metadata">

**Author:** ![elastic\_paul](https://avatars.discourse-cdn.com/v4/letter/e/ebca7d/32.png) [@elastic\_paul](https://discuss.elastic.co/u/elastic_paul)\
**Post date:** [November 27, 2017, 11:31am UTC](https://discuss.elastic.co/t/using-my-own-document-id-is-there-a-faster-way/108667/21 "2017-11-27T11:31:35Z")

</div>

This is the config from my old logstash. As I said, I have recently upgraded to 6.0 and I wanted to know if this was still the way to do it.  
The answer seems to be no! I have looked at fingerprint (which came in at 5.4) and it looks the way to do it so I will change my logstash to use this new plugin.  
Thanks

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 27, 2017, 11:34am UTC](https://discuss.elastic.co/t/using-my-own-document-id-is-there-a-faster-way/108667/22 "2017-11-27T11:34:11Z")

</div>

The fingerprint filter has been around for a very long time and is not new. As you can see [from this blog post](https://www.elastic.co/blog/logstash-lessons-handling-duplicates), this is still the way to do duplicate prevention.

---

<div class="post-metadata">

**Author:** ![elastic\_paul](https://avatars.discourse-cdn.com/v4/letter/e/ebca7d/32.png) [@elastic\_paul](https://discuss.elastic.co/u/elastic_paul)\
**Post date:** [November 27, 2017, 12:46pm UTC](https://discuss.elastic.co/t/using-my-own-document-id-is-there-a-faster-way/108667/23 "2017-11-27T12:46:49Z")

</div>

Does this look the right way to do it?

```auto
    fingerprint {
      key => "secret"
      method => "SHA1"
      source => ["ip", "fingerprint_sha1"]
      concatenate_sources => true
      target => "[@metadata][fingerprint]"
    }

output {
      elasticsearch {
        user => xx
        password => xx
        action => "index"
        document_id => "%{[@metadata][fingerprint]}"

```

I am seeing deprecation warnings but I am not clear if they are related.

```auto
17-11-27T12:41:24,700][WARN][o.e.d.i.m.UidFieldMapper] Fielddata access on the _uid field is deprecated, use _id instead

```

---

<div class="post-metadata">

**Author:** ![elastic\_paul](https://avatars.discourse-cdn.com/v4/letter/e/ebca7d/32.png) [@elastic\_paul](https://discuss.elastic.co/u/elastic_paul)\
**Post date:** [November 27, 2017, 1:58pm UTC](https://discuss.elastic.co/t/using-my-own-document-id-is-there-a-faster-way/108667/24 "2017-11-27T13:58:34Z")

</div>

This is not working as I thought it would?  
From the ES index I have this:

```auto
ip = 209.212.33.52
fingerprint_sha1 = b82d0ec6825ba8ee89220d2a609dc851a2797027
_id = 2bd3142564d0d83691b3bb014e6404a8

```

But I was expecting \_id to be the document\_id? So shouldn't that be a SHA1 value? Effectively SHA1(ip + fingerprint\_sha1)

Am I misunderstanding this completely?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 27, 2017, 2:52pm UTC](https://discuss.elastic.co/t/using-my-own-document-id-is-there-a-faster-way/108667/25 "2017-11-27T14:52:25Z")

</div>

That look like a SHA1 hash is used for the \_id, so looks fine as far as I can see.

---

<div class="post-metadata">

**Author:** ![elastic\_paul](https://avatars.discourse-cdn.com/v4/letter/e/ebca7d/32.png) [@elastic\_paul](https://discuss.elastic.co/u/elastic_paul)\
**Post date:** [November 29, 2017, 6:24am UTC](https://discuss.elastic.co/t/using-my-own-document-id-is-there-a-faster-way/108667/26 "2017-11-29T06:24:54Z")

</div>

Over on a separate thread in [Elasticsearch](https://discuss.elastic.co/t/how-to-start-two-instances-of-es-6-0-as-a-service/109328/4) I discussed and implemented 2 instances per node in my cluster, along with many of the other changes discussed here.

I also changed my logstash configuration so that all data is sent to a single ES instance, and that instance can then pass the data within the cluster. Rather than flooding all 8 instances (2 per node) with data, which resulted in overloading everything.

Its all stable and loading in data in the 12,000-15,000/s range.  
Thanks for all of the help and guidance.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 27, 2017, 6:24am UTC](https://discuss.elastic.co/t/using-my-own-document-id-is-there-a-faster-way/108667/27 "2017-12-27T06:24:56Z")

</div>

Closed on December 27, 2017, 6:24am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

[Previous page](https://discuss.elastic.co/t/using-my-own-document-id-is-there-a-faster-way/108667.md?page=1)