# Elastic document\_id **URL:** \ **Category:** Logstash\ **Created:** [March 28, 2023, 5:39pm UTC](https://discuss.elastic.co/t/elastic-document-id/328738 "2023-03-28T17:39:37Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![bmagistro1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bmagistro1/32/118411_2.png) [@bmagistro1](https://discuss.elastic.co/u/bmagistro1)\ **Post date:** [March 28, 2023, 5:39pm UTC](https://discuss.elastic.co/t/elastic-document-id/328738/1 "2023-03-28T17:39:37Z")
Is there any defined behavior for document\_id ([Elasticsearch output plugin | Logstash Reference [8.6] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-document_id)) similar to pipeline ([Elasticsearch output plugin | Logstash Reference [8.6] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-pipeline))? We have at least one data source where we have chosen to set a document id (was started years ago so don't know/remember the history on why) but we have been cleaning up our pipeline and were wondering if we could leverage a metadata field to specify the document\_id but if empty it would behave the same as if the document\_id wasn't specified. We are doing this with pipeline and it has helped greatly simplify the output logic, other than the document\_id being present the output blocks are the same. Any thoughts? Current ```auto if source { elasticsearch { ... document_id => "" } } else { elasticsearch { ... } } ``` ---
**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Post date:** [March 28, 2023, 5:56pm UTC](https://discuss.elastic.co/t/elastic-document-id/328738/2 "2023-03-28T17:56:04Z")
if you use metadata field to define \_id and if that is blank then you won't have that document in your index. or should I say it will overwrite older doc. for example first record has metadata\_id = "ben" and you have one record second record it is empty and you will have that record inserted in index with \_id = metadata\_id now third record is empty then it will overwrite second record with third. ---
**Author:** ![bmagistro1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bmagistro1/32/118411_2.png) [@bmagistro1](https://discuss.elastic.co/u/bmagistro1)\ **Post date:** [March 28, 2023, 6:38pm UTC](https://discuss.elastic.co/t/elastic-document-id/328738/3 "2023-03-28T18:38:25Z")
That's how I interpreted things too. Wanted to double check if there was something I was missing or someway to have it behave closer to that of pipeline. ex record1 has id 1231, gets inserted with 1231 record2 has id "", gets created without specifying id record 3 has id "", gets created without specifying id, does not overwrite record 2 either record2 has id 1231; follows settings of doc\_as\_upsert + action (or any other parameters I'm not thinking of) which may overwrite previous record ---
**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Post date:** [March 28, 2023, 7:17pm UTC](https://discuss.elastic.co/t/elastic-document-id/328738/4 "2023-03-28T19:17:30Z")
> [@bmagistro1](#): > > ex > record1 has id 1231, gets inserted with 1231 > record2 has id "", gets created without specifying id > record 3 has id "", gets created without specifying id, does not overwrite record 2 either > record2 has id 1231; follows settings of doc\_as\_upsert + action (or any other parameters I'm not thinking of) which may overwrite previous record record1 is simple record2 you will have to create ID it it is null record3 again you have to create random id if it null and it won't overwrite record3 if it has same id as record1 then it will automatically overwrite previous record. you don't have to do anything. consider \_id has primary uniq key in relational database. and it has to be uniq. ---
**Author:** ![bmagistro1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bmagistro1/32/118411_2.png) [@bmagistro1](https://discuss.elastic.co/u/bmagistro1)\ **Post date:** [March 28, 2023, 8:04pm UTC](https://discuss.elastic.co/t/elastic-document-id/328738/5 "2023-03-28T20:04:36Z")
Agreed on the primary key function. If I don't include `document_id` in the output config for elastic, it will generate an id on its own and I don't need to do anything. What I'm looking for is if there is a "meta value" or a way to have "" (empty string) trigger the generation that happens today when `document_id` is not included in the output config for elastic. I guess the alternative is to implement the same generation function early on and have events override that value when they need to. This would let some events provide a an ID when they know they need to. My concern with trying to implement the id function is collisions. ---
**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Post date:** [March 30, 2023, 4:08pm UTC](https://discuss.elastic.co/t/elastic-document-id/328738/6 "2023-03-30T16:08:51Z")
there is meta field called "metadata id" this is how you generate that on logstash `mutate { add_field => { "[@metadata][id]" => "%{[host][name]}_%{some_field}" } }` after this you can check if it is blank then reset to whatever you like and on output section you can use it like `document_id => "%{[@metadata][id]}"` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 27, 2023, 4:09pm UTC](https://discuss.elastic.co/t/elastic-document-id/328738/7 "2023-04-27T16:09:26Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.