# How to create my own document\_id in logstash?

**URL:** <https://discuss.elastic.co/t/how-to-create-my-own-document-id-in-logstash/1416>\
**Category:** Logstash\
**Created:** [May 27, 2015, 6:42pm UTC](https://discuss.elastic.co/t/how-to-create-my-own-document-id-in-logstash/1416 "2015-05-27T18:42:30Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![elastic\_paul](https://avatars.discourse-cdn.com/v4/letter/e/ebca7d/32.png) [@elastic\_paul](https://discuss.elastic.co/u/elastic_paul)\
**Post date:** [May 27, 2015, 6:42pm UTC](https://discuss.elastic.co/t/how-to-create-my-own-document-id-in-logstash/1416/1 "2015-05-27T18:42:30Z")

</div>

I would like to create my own document\_id to avoid duplication.

I would like to make the document\_id as an MD5 hash of two fields; "ip" and "sha1\_fingerprint".

eg; in pseudo code:

md5\_hex( "ip" + " sha1\_fingerprint" )

Thanks

---

<div class="post-metadata">

**Author:** ![suyograo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/suyograo/32/44898_2.png) [@suyograo](https://discuss.elastic.co/u/suyograo)\
**Post date:** [May 28, 2015, 1:12am UTC](https://discuss.elastic.co/t/how-to-create-my-own-document-id-in-logstash/1416/2 "2015-05-28T01:12:32Z")

</div>

You can first inject a field called `computed_id` using a ruby filter: [https://www.elastic.co/guide/en/logstash/current/plugins-filters-ruby.html](https://www.elastic.co/guide/en/logstash/current/plugins-filters-ruby.html)

So, in ruby filter you can do: `event[computed_id] => Digest::MD5.hexdigest('event[ip] + sha1')`. After assigning this to a `computed_field` you can use it in ES output `document_id => "%{computed_field}"`

---

<div class="post-metadata">

**Author:** ![elastic\_paul](https://avatars.discourse-cdn.com/v4/letter/e/ebca7d/32.png) [@elastic\_paul](https://discuss.elastic.co/u/elastic_paul)\
**Post date:** [May 28, 2015, 5:49am UTC](https://discuss.elastic.co/t/how-to-create-my-own-document-id-in-logstash/1416/3 "2015-05-28T05:49:24Z")

</div>

Thanks very much for this. Just what I wanted. Just to help anyone finding this question, here is the code I actually used:

```
ruby {
  code => "require 'digest/md5';
  event['computed_id'] = Digest::MD5.hexdigest(event['ip'] + event['sha1_fingerprint'])"
}

```

Then

```
document_id => "%{computed_id}"

```

Can I ask two more related questions:

1. How can I remove the field after I have set the document\_id? I don't want it in my stored data. eg; remove event['computed\_id']

2. Its seems that my index is BIGGER doing it this way? Any ideas? I thought that deduplication would save space? It can't be because of the extra 'computed\_id' field can it?

Thanks

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [May 28, 2015, 7:02am UTC](https://discuss.elastic.co/t/how-to-create-my-own-document-id-in-logstash/1416/4 "2015-05-28T07:02:41Z")

</div>

> 1. How can I remove the field after I have set the document\_id? I don't want it in my stored data. eg; remove event['computed\_id']

If you're using Logstash 1.5 you can store the computed id field as a subfield of the [@metadata field](https://www.elastic.co/guide/en/logstash/current/configuration.html#metadata). None of those fields propagate to outputs.

> 1. Its seems that my index is BIGGER doing it this way? Any ideas? I thought that deduplication would save space? It can't be because of the extra 'computed\_id' field can it?

Well, to what extent are you actually deduplicating events? The computed id field (obviously) only have unique values so they'll add a lot of terms to the index.

---

<div class="post-metadata">

**Author:** ![elastic\_paul](https://avatars.discourse-cdn.com/v4/letter/e/ebca7d/32.png) [@elastic\_paul](https://discuss.elastic.co/u/elastic_paul)\
**Post date:** [May 28, 2015, 7:52am UTC](https://discuss.elastic.co/t/how-to-create-my-own-document-id-in-logstash/1416/5 "2015-05-28T07:52:26Z")

</div>

> If you're using Logstash 1.5 you can store the computed id field as a subfield of the @metadata field. None of those fields propagate to outputs.

That is exactly the function I am after thank you. Can you just help me with the syntax though?

I currently use:

```
event['computed_id'] = Digest::MD5.hexdigest(event['ip'] + event['sha1_fingerprint'])

```

So will that become:

```
@metadata[md5] = Digest::MD5.hexdigest(event['ip'] + event['sha1_fingerprint'])

```

That doesn't seem to work - do I need to do a mutate to add @metadata[md5] first? I don't find the syntax very easy.  
Thanks again

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [May 28, 2015, 9:15am UTC](https://discuss.elastic.co/t/how-to-create-my-own-document-id-in-logstash/1416/6 "2015-05-28T09:15:55Z")

</div>

The document ID in ES is going to be pretty damn unique anyway, you might be reinventing the wheel here!

---

<div class="post-metadata">

**Author:** ![elastic\_paul](https://avatars.discourse-cdn.com/v4/letter/e/ebca7d/32.png) [@elastic\_paul](https://discuss.elastic.co/u/elastic_paul)\
**Post date:** [May 28, 2015, 9:47am UTC](https://discuss.elastic.co/t/how-to-create-my-own-document-id-in-logstash/1416/7 "2015-05-28T09:47:03Z")

</div>

HI, I am trying to deduplicate the records.

I have lots of IP + SHA1\_fingerprints that are all the same apart from timestamp, and I don't want or need them. So my thought was to use a hash of (IP + SHA1\_fingerprint) as the doc\_id to only save one of the records. Does that make sense?  
eg;  
time1 IP1 SHA1\_1  
time2 IP1 SHA1\_1  
time3 IP1 SHA1\_1

- Don't need or want 3 docs so only save one ?

Meanwhile I am completely stuck on trying to make use the @metadata field with the ruby code. Could someone be kind enough to help a beginner out and change my posted code to use @metafield? I just can not get the syntax.

Thanks

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [May 29, 2015, 9:43am UTC](https://discuss.elastic.co/t/how-to-create-my-own-document-id-in-logstash/1416/8 "2015-05-29T09:43:57Z")

</div>

> `@metadata[md5] = Digest::MD5.hexdigest(event['ip'] + event['sha1_fingerprint'])`

`@metadata` is a normal message field (except that it doesn't propagate to outputs) so this is what you're looking for:

```
event['@metadata']['md5'] = Digest::MD5.hexdigest(event['ip'] + event['sha1_fingerprint'])

```

---

<div class="post-metadata">

**Author:** ![elastic\_paul](https://avatars.discourse-cdn.com/v4/letter/e/ebca7d/32.png) [@elastic\_paul](https://discuss.elastic.co/u/elastic_paul)\
**Post date:** [May 29, 2015, 10:56am UTC](https://discuss.elastic.co/t/how-to-create-my-own-document-id-in-logstash/1416/9 "2015-05-29T10:56:18Z")

</div>

Perfect thanks.

I also had to edit the elasticsearch plugin to accept an HTTP code 409 when using the create command. See my other thread for that one.

> [@Re: "create action does not use \_id for request #79"](https://discuss.elastic.co/t/re-create-action-does-not-use--id-for-request-79/1501):
>
> create action does not use \_id for request #79 hxxps://github.com/logstash-plugins/logstash-output-elasticsearch/issues/79 This means that I can not use my own deduplication where I manually set the document\_id to a given hash. My document\_id will not be the same as the \_id that is being used and so it won't work with action=\>create Have I got that right? EDIT: NO! I did not get this right. It failed because ES returns an HTTP code 409 for the duplicate (correctly) but the plugin fails on th…

---

<div class="post-metadata">

**Author:** ![tomr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tomr/32/48260_2.png) [@tomr](https://discuss.elastic.co/u/tomr)\
**Post date:** [June 26, 2015, 10:02am UTC](https://discuss.elastic.co/t/how-to-create-my-own-document-id-in-logstash/1416/10 "2015-06-26T10:02:32Z")

</div>

Is there a good reason not to use the computed hash as the document \_id ?

I'm looking at doing the same thing, and IIUC using the hash as \_id would allow me to use op\_type to reduce the cost of a log replay.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [June 26, 2015, 10:05am UTC](https://discuss.elastic.co/t/how-to-create-my-own-document-id-in-logstash/1416/11 "2015-06-26T10:05:46Z")

</div>

Depends, why would you want to replay?

---

<div class="post-metadata">

**Author:** ![tomr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tomr/32/48260_2.png) [@tomr](https://discuss.elastic.co/u/tomr)\
**Post date:** [June 26, 2015, 10:13am UTC](https://discuss.elastic.co/t/how-to-create-my-own-document-id-in-logstash/1416/12 "2015-06-26T10:13:35Z")

</div>

In short, unreliable log-delivery.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [June 26, 2015, 10:19am UTC](https://discuss.elastic.co/t/how-to-create-my-own-document-id-in-logstash/1416/13 "2015-06-26T10:19:10Z")

</div>

Might be easier to do this in your other thread 🙂

---

<div class="post-metadata">

**Author:** ![tomr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tomr/32/48260_2.png) [@tomr](https://discuss.elastic.co/u/tomr)\
**Post date:** [June 26, 2015, 11:08am UTC](https://discuss.elastic.co/t/how-to-create-my-own-document-id-in-logstash/1416/14 "2015-06-26T11:08:01Z")

</div>

I hope you appreciate the irony 😄

For reference, this now works for me.  
in the filter section:

```
ruby {
  code => "require 'digest/md5';
  event['@metadata']['computed_id'] = Digest::MD5.hexdigest(event['message'])"
}

```

and in my ES output section:

```
document_id => "%{[@metadata][computed_id]}"

```

thanks all  
t

---

<div class="post-metadata">

**Author:** ![fninja](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fninja/32/6293_2.png) [@fninja](https://discuss.elastic.co/u/fninja)\
**Post date:** [December 1, 2015, 10:07am UTC](https://discuss.elastic.co/t/how-to-create-my-own-document-id-in-logstash/1416/15 "2015-12-01T10:07:35Z")

</div>

this so needs to be come a plugin.  
@warcolm  
how can one use the logstash-generated document id to avoid duplication, if it gets generated anew every time you index a document?

So if you index "document a" 3 times it will receive a different document id upon each time.

Now if you create a hash out of "document a" three times, this hash will be the same three times.  
Great feature for replaying logs gracefully.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [December 1, 2015, 10:28am UTC](https://discuss.elastic.co/t/how-to-create-my-own-document-id-in-logstash/1416/16 "2015-12-01T10:28:16Z")

</div>

There are already two plugins that you can use for this and avoid having to use the Ruby filter:

The [checksum filter plugin](https://www.elastic.co/guide/en/logstash/current/plugins-filters-checksum.html) is labelled as experimental, but allows you to specify which fields that you want to include in the hash calculation, so you could choose to exclude the timestamp field as in your example.

The [fingerprint filter plugin](https://www.elastic.co/guide/en/logstash/current/plugins-filters-fingerprint.html) seems less experimental, but only supports specifying a single field as input. You would therefore need to concatenate the fields you want to hash into a single field, possibly under '@metadata' before running this.

---

<div class="post-metadata">

**Author:** ![sjivan](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@sjivan](https://discuss.elastic.co/u/sjivan)\
**Post date:** [November 17, 2016, 10:30pm UTC](https://discuss.elastic.co/t/how-to-create-my-own-document-id-in-logstash/1416/17 "2016-11-17T22:30:46Z")

</div>

Hi,  
What's the LS 5.0 equivalent of the following since direct access of event fields is not longer allowed?

`event['@metadata']['myfield'] = 'foo'`

I tried

`event.get('@metadata').set('myfield', 'foo')`

but I get the following exception

`Ruby exception occurred: undefined method`set' for #Hash:0x4a52075d`

Thanks,  
Sanjiv

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [November 18, 2016, 6:30am UTC](https://discuss.elastic.co/t/how-to-create-my-own-document-id-in-logstash/1416/18 "2016-11-18T06:30:44Z")

</div>

See [https://www.elastic.co/guide/en/logstash/current/event-api.html#\_event\_api](https://www.elastic.co/guide/en/logstash/current/event-api.html#_event_api). Please start a new thread if you have any follow-up questions.

---

<div class="post-metadata">

**Author:** ![EthanStark](https://avatars.discourse-cdn.com/v4/letter/e/d6d6ee/32.png) [@EthanStark](https://discuss.elastic.co/u/EthanStark)\
**Post date:** [June 7, 2017, 11:19am UTC](https://discuss.elastic.co/t/how-to-create-my-own-document-id-in-logstash/1416/19 "2017-06-07T11:19:05Z")

</div>

output {  
if [document\_id] {  
elasticsearch\_http {  
host =\> "127.0.0.1"  
document\_id =\> "%{document\_id}"  
}  
} else {  
elasticsearch\_http {  
host =\> "127.0.0.1"  
}  
}  
}

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 4, 2022, 4:00am UTC](https://discuss.elastic.co/t/how-to-create-my-own-document-id-in-logstash/1416/20 "2022-11-04T04:00:12Z")

</div>