# Update Existing document through logstash

**URL:** <https://discuss.elastic.co/t/update-existing-document-through-logstash/329077>\
**Category:** Logstash\
**Created:** [March 31, 2023, 6:49pm UTC](https://discuss.elastic.co/t/update-existing-document-through-logstash/329077 "2023-03-31T18:49:32Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![rubhamra](https://avatars.discourse-cdn.com/v4/letter/r/858c86/32.png) [@rubhamra](https://discuss.elastic.co/u/rubhamra)\
**Post date:** [March 31, 2023, 6:49pm UTC](https://discuss.elastic.co/t/update-existing-document-through-logstash/329077/1 "2023-03-31T18:49:32Z")

</div>

logstash pipeline is not updating existing document for the same id, I have couples of fields which got updated frequestly for example Last Modified Date.

I have below logstash config.

```auto
output {		
   elasticsearch {
        id => "api-main"
        hosts => ["https://localhost:9200"]
        #ssl => true
        #cacert => "/usr/share/logstash/certs/http_ca.crt"
        cacert => '/etc/logstash/config/certs/ca.crt'
        user => "elastic"
        password => "xxxxxxx"
	    index => "change-dev-%{+YYYY-MM-dd}"
	    #document_id => "%{[response][body][entries][values][Request_ID]}"
        document_id => "%{[response][body][entries][values][Change_ID]}"
        #document_id => "%{[@metadata][_id]}"
        action => "update"
        doc_as_upsert => true
        #data_stream_sync_fields => true
	}
    stdout {
        codec => rubydebug {
            metadata => true
        }
    }
}

```

If I use Action =\> create then the index is created matching index template using Data Stream,  
but then later if there is an update in the field for example ('Last Modified date' field is changed ) but logstash doesn't update that field in elasticsearch . I tried to use update and doc\_as\_upsert but it's not working.

I am also getting below exception in logs.

```auto
[WARN] 2023-03-31 18:37:16.716 [[main]>worker0] elasticsearch - Could not index event to Elasticsearch. status: 400, action: ["update", {:_id=>"CRQ000000000081", :_index=>"change-dev-2023-03-31", :routing=>nil, :retry_on_conflict=>1},
"status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"only write ops with an op_type of create are allowed in data streams"}}}

```

Please suggest.

---

<div class="post-metadata">

**Author:** ![jba](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jba/32/118482_2.png) [@jba](https://discuss.elastic.co/u/jba)\
**Post date:** [March 31, 2023, 8:10pm UTC](https://discuss.elastic.co/t/update-existing-document-through-logstash/329077/2 "2023-03-31T20:10:05Z")

</div>

The index you are trying to update a document in, is it an index or is it a data stream?

---

<div class="post-metadata">

**Author:** ![rubhamra](https://avatars.discourse-cdn.com/v4/letter/r/858c86/32.png) [@rubhamra](https://discuss.elastic.co/u/rubhamra)\
**Post date:** [March 31, 2023, 8:10pm UTC](https://discuss.elastic.co/t/update-existing-document-through-logstash/329077/3 "2023-03-31T20:10:53Z")

</div>

It's a data stream

---

<div class="post-metadata">

**Author:** ![jba](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jba/32/118482_2.png) [@jba](https://discuss.elastic.co/u/jba)\
**Post date:** [March 31, 2023, 8:24pm UTC](https://discuss.elastic.co/t/update-existing-document-through-logstash/329077/4 "2023-03-31T20:24:29Z")

</div>

Yeah, well, there is your problem (I think). It is my understanding (which could be wrong) that indexes are for documents and data that you can change/update/delete documents in, while data streams are for write-once/read-many documents.

If my understanding is correct, then the error message you get makes sense: You are only allowed to create documents - not update (or upsert = insert-or-update).

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [March 31, 2023, 8:40pm UTC](https://discuss.elastic.co/t/update-existing-document-through-logstash/329077/5 "2023-03-31T20:40:21Z")

</div>

> [@rubhamra](#):
>
> It's a data stream

You cannot update a data stream through Logstash, if you need to do this then you need to use normal time-based indices.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 28, 2023, 8:41pm UTC](https://discuss.elastic.co/t/update-existing-document-through-logstash/329077/6 "2023-04-28T20:41:02Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.