# Logstash - how to overwrite document instead of creating new ones

**URL:** <https://discuss.elastic.co/t/logstash-how-to-overwrite-document-instead-of-creating-new-ones/191456>\
**Category:** Logstash\
**Created:** [July 19, 2019, 10:56pm UTC](https://discuss.elastic.co/t/logstash-how-to-overwrite-document-instead-of-creating-new-ones/191456 "2019-07-19T22:56:08Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![mickey\_lubs](https://avatars.discourse-cdn.com/v4/letter/m/d78d45/32.png) [@mickey\_lubs](https://discuss.elastic.co/u/mickey_lubs)\
**Post date:** [July 19, 2019, 10:56pm UTC](https://discuss.elastic.co/t/logstash-how-to-overwrite-document-instead-of-creating-new-ones/191456/1 "2019-07-19T22:56:08Z")

</div>

Hi Guys,

I need help. I have got db2 logstash pipeline which runs a query at intervals. The problem is everytime it runs the query, it creates a new document with same data which fills up my index with duplicate data. How can i overwrite all document\_id everytime my query runs ?

---

<div class="post-metadata">

**Author:** ![wangqinghuan](https://avatars.discourse-cdn.com/v4/letter/w/d26b3c/32.png) [@wangqinghuan](https://discuss.elastic.co/u/wangqinghuan)\
**Post date:** [July 20, 2019, 9:38am UTC](https://discuss.elastic.co/t/logstash-how-to-overwrite-document-instead-of-creating-new-ones/191456/2 "2019-07-20T09:38:34Z")

</div>

You need to generate your own document\_id and elasticsearch will use document\_id to update document.

> ```
> output {
> elasticsearch {
> host => 'your es host'
> action => 'update'
> document_id => 'your generated document id'
> index => 'your index name
> }
> }
> 
> ```

For your situation, fill document\_id with a id field from db2 can overwrite document.

> ```
> output {
> elasticsearch {
> host => 'your es host'
> action => 'update'
> document_id => "%{[id]}"
> index => 'your index name
> }
> }
> 
> ```

---

<div class="post-metadata">

**Author:** ![mickey\_lubs](https://avatars.discourse-cdn.com/v4/letter/m/d78d45/32.png) [@mickey\_lubs](https://discuss.elastic.co/u/mickey_lubs)\
**Post date:** [July 20, 2019, 9:56pm UTC](https://discuss.elastic.co/t/logstash-how-to-overwrite-document-instead-of-creating-new-ones/191456/3 "2019-07-20T21:56:53Z")

</div>

Thanks but below ia error i get with the proposed settings:

Here are the settings im using:

filter {  
fingerprint {  
method =\> "UUID"  
target =\> "[@metadata][uuid]"  
}

mutate {  
add\_field =\> { "uuid" =\> "%{[@metadata][uuid]}" }  
}

}  
output {  
elasticsearch {  
hosts =\> ["localhost:9200"]  
action =\> "update"  
document\_id =\> "%{uuid}"  
index =\> "storageg-monitoring-%{+YYYY.MM.dd}"  
doc\_as\_upsert =\> false  
}  
}

=================================================  
Error:  
[2019-07-20T23:52:08,097][WARN][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=\>404, :action=\>["update", {:\_id=\>"2c18be01-d143-4511-9c54-208c4f57b897", :\_index=\>"storageg-monitoring-2019.07.20", :\_type=\>"\_doc", :routing=\>nil, :retry\_on\_conflict=\>1}, #LogStash::Event:0x5f69d037], :response=\>{"update"=\>{"\_index"=\>"storageg-monitoring-2019.07.20", "\_type"=\>"\_doc", "\_id"=\>"2c18be01-d143-4511-9c54-208c4f57b897", "status"=\>404, "error"=\>{"type"=\>"document\_missing\_exception", "reason"=\>"[\_doc][2c18be01-d143-4511-9c54-208c4f57b897]: document missing", "index\_uuid"=\>"PHkRN-fTQa2MRoOzn5P5-g", "shard"=\>"0", "index"=\>"storageg-monitoring-2019.07.20"}}}}

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 21, 2019, 12:07pm UTC](https://discuss.elastic.co/t/logstash-how-to-overwrite-document-instead-of-creating-new-ones/191456/4 "2019-07-21T12:07:17Z")

</div>

I believe that is telling you that the document you are trying to update does not exist, so you are actually trying to create a new document. If that is what you want to do set doc\_as\_upsert to be true.

---

<div class="post-metadata">

**Author:** ![ylasri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ylasri/32/86120_2.png) [@ylasri](https://discuss.elastic.co/u/ylasri)\
**Post date:** [July 21, 2019, 12:43pm UTC](https://discuss.elastic.co/t/logstash-how-to-overwrite-document-instead-of-creating-new-ones/191456/5 "2019-07-21T12:43:12Z")

</div>

Use this filter to generate a unique id for your event based on fields that define the unicity of the document

```
fingerprint {
	method => "MD5"
	key => "xxxxxxxxx"
	concatenate_sources => true
	concatenate_all_fields => false
	source => ["field1", ""field2"]
	target => "[@metadata][document_id]"
}

output {
	elasticsearch {
	hosts => ["http://localhost:9200"]
	user => "xxx"
	password => "xxxx"
	index => "index-name%{+YYYY-MM-dd}"
	document_id => "%{[@metadata][document_id]}"
}
}
```

---

<div class="post-metadata">

**Author:** ![mickey\_lubs](https://avatars.discourse-cdn.com/v4/letter/m/d78d45/32.png) [@mickey\_lubs](https://discuss.elastic.co/u/mickey_lubs)\
**Post date:** [July 21, 2019, 10:39pm UTC](https://discuss.elastic.co/t/logstash-how-to-overwrite-document-instead-of-creating-new-ones/191456/6 "2019-07-21T22:39:50Z")

</div>

Thanks so much guys!  
It works well with your proposed settings 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 18, 2019, 10:39pm UTC](https://discuss.elastic.co/t/logstash-how-to-overwrite-document-instead-of-creating-new-ones/191456/7 "2019-08-18T22:39:51Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.