# How to set "\_id" value in elasticsearch document as my custom document id **URL:** \ **Category:** Beats\ **Tags:** beats-development\ **Created:** [August 16, 2018, 7:36am UTC](https://discuss.elastic.co/t/how-to-set-id-value-in-elasticsearch-document-as-my-custom-document-id/144644 "2018-08-16T07:36:52Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![r.ganeshbabu](https://avatars.discourse-cdn.com/v4/letter/r/4da419/32.png) [@r.ganeshbabu](https://discuss.elastic.co/u/r.ganeshbabu)\ **Post date:** [August 16, 2018, 7:36am UTC](https://discuss.elastic.co/t/how-to-set-id-value-in-elasticsearch-document-as-my-custom-document-id/144644/1 "2018-08-16T07:36:53Z")
Hi All, We are developing a customized beats for the monitoring tool and we are capturing metrics like cpu, memory, disk etc.. Data was sending it through output logstash which is configured in beat.yml file as like below, ``` #----------------------------- Logstash output -------------------------------- output.logstash: hosts: ["localhost:5044"] ``` We are receiving fields from the monitoring tool and one of the field as **test\_serial\_id** and I want to set the test\_serial\_id as "\_id" in the elasticsearch document in order to remove the duplicates. How to achieve this in beats? Please share your thoughts and it would be very helpful. Thanks, Ganeshbabu R ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [August 16, 2018, 12:58pm UTC](https://discuss.elastic.co/t/how-to-set-id-value-in-elasticsearch-document-as-my-custom-document-id/144644/2 "2018-08-16T12:58:22Z")
You can generate an ID and pubish it by setting adding the ID to your events meta data: ```auto event := beat.Event{ Timestamp: ... Fields: ... Meta: common.MapStr{"id": } } ``` When publishing to elasticsearch, the id value will be used for `_id`. When publishing to logstash/redis/kafka, beats add a `@metadata` field to the event. This field will contain the id. You can configure the elasticsearch output in Logstash to use `[@metadata][id]`. When indexing, elasticsearch accepts an operation type. If you set it to 'create', then duplicate entries will be detected, but not overwritten. If you don't set it to create, then the old entry will be marked as deleted and the new event will be indexed. ---
**Author:** ![r.ganeshbabu](https://avatars.discourse-cdn.com/v4/letter/r/4da419/32.png) [@r.ganeshbabu](https://discuss.elastic.co/u/r.ganeshbabu)\ **Post date:** [August 16, 2018, 1:50pm UTC](https://discuss.elastic.co/t/how-to-set-id-value-in-elasticsearch-document-as-my-custom-document-id/144644/3 "2018-08-16T13:50:43Z")
Thanks for sharing your inputs @steffens > [@steffens](#): > > You can configure the elasticsearch output in Logstash to use `[@metadata][id]` But we configured beats port in .yml file and sending data to port 5044 via **output.logstash**. Can I send the document id through output.logstash? Let me know your thoughts. Regards, Ganeshbabu R ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [August 16, 2018, 6:12pm UTC](https://discuss.elastic.co/t/how-to-set-id-value-in-elasticsearch-document-as-my-custom-document-id/144644/4 "2018-08-16T18:12:35Z")
> Can I send the document id through output.logstash? Yes. Please read this paragraph again: > When publishing to logstash/redis/kafka, beats add a @metadata field to the event. This field will contain the id. You can configure the elasticsearch output in Logstash to use [@metadata][id]. You can test and inspect events in LS via: ```auto input { beat { ... } } output { stdout { codec => rubydebug { metadata => true } } } ``` This will print all events, include the `@metadata` section, which should contain the `id` field. If it's missing, then you need to update libbeat, or you didn't add it to your event in the beat. ---
**Author:** ![r.ganeshbabu](https://avatars.discourse-cdn.com/v4/letter/r/4da419/32.png) [@r.ganeshbabu](https://discuss.elastic.co/u/r.ganeshbabu)\ **Post date:** [August 17, 2018, 8:18am UTC](https://discuss.elastic.co/t/how-to-set-id-value-in-elasticsearch-document-as-my-custom-document-id/144644/5 "2018-08-17T08:18:40Z")
Hi @steffens, > [@steffens](#): > > This will print all events, include the `@metadata` section, which should contain the `id` field. If it's missing, then you need to update libbeat, or you didn't add it to your event in the beat. We tried printing all the events and we did find the metadata section in the result and below is the code snippet we have added in event beat, ``` event := beat.Event{ Timestamp: time.Unix((i.Timestamp/1000), 0), Fields:common.MapStr{ "type" :"traversebeat", "TestSerialNumber": TestSerialNumber, "DeviceSerialNumber": DeviceSerialNumber, "AccountSerialNumber": AccountSerialNumber, }, Meta: common.MapStr{"id": doc_id}, } ``` and below is the printed event from the logstash, and I can see that @metadata "custom id" value is present, ``` { "type" => "testbeat", "DeviceSerialNumber" => 6020621, "AccountSerialNumber" => 5730000, "TestSerialNumber" => 6021700, "MinValue" => 446521, "beat" => { "hostname" => "localhost", "name" => "localhost", "version" => "6.2.0" }, "@metadata" => { "beat" => "testbeat", "id" => "SCO-SRSWEB-PRD1-1534493100000-6021700", "type" => "doc", "version" => "6.2.0" }, } ``` But when I sent the same data to beats port 5044 in logstash the **\_id** value is not picking the same as above. Its creating a generated ID in the document. Kindly share your thoughts. Regards, Ganeshbabu R ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [August 17, 2018, 12:31pm UTC](https://discuss.elastic.co/t/how-to-set-id-value-in-elasticsearch-document-as-my-custom-document-id/144644/6 "2018-08-17T12:31:50Z")
Beats just forward some info via `@metadata`. Beats don't force you to make use of these information. You have to configure the Elasticsearch output in Logstash to use the id. Check out the [Logstash elasticsearch outputs docs](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html). Settings you might be interested in: action, document\_id. Configure `action => "create"` and `document_id => "[@metadata][id]"`. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 14, 2018, 12:31pm UTC](https://discuss.elastic.co/t/how-to-set-id-value-in-elasticsearch-document-as-my-custom-document-id/144644/7 "2018-09-14T12:31:52Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.