# How to progressively update my document

**URL:** <https://discuss.elastic.co/t/how-to-progressively-update-my-document/36575>\
**Category:** Logstash\
**Created:** [December 7, 2015, 9:04pm UTC](https://discuss.elastic.co/t/how-to-progressively-update-my-document/36575 "2015-12-07T21:04:30Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![jjdepaul](https://avatars.discourse-cdn.com/v4/letter/j/e0b2c6/32.png) [@jjdepaul](https://discuss.elastic.co/u/jjdepaul)\
**Post date:** [December 7, 2015, 9:04pm UTC](https://discuss.elastic.co/t/how-to-progressively-update-my-document/36575/1 "2015-12-07T21:04:30Z")

</div>

I have LS 2.0.x ES2.0 Shield 2.0 and Kibana 4.2

My configuration reads incoming application log data. The data represents all of the work flow events that our orders go thorugh - a series of pre-defined steps (approx 7 steps) that arrive at different times.

I want my Index, to contain only one document for each order. I have created in my mappings individual fields (buckets) in that index that keep track of the different work flow steps. Thus, everytime a new event comes in for that order, I want to update a different "bucket" in the document to reflect the completion of that step. Thus WFStep1 timestamp, WFStep2 timestamp, WFStep3...

I control the document\_id to facilitate the updates so I get one document per order.

Thus I want to populate the document gradually as the different WorkFlow steps are completed for each order. At the end of all WF steps I want all of the 'buckets'to be populated as a result.

It almost works... the problem is that the final document only contains the very last WorkFlow step update and doesn't 'remember' any other steps that happened before it. Even though there were 7 transactions for that same order only the last one is retained in the document. Why aren't the other fields populated by prior updates preserved in the document? How can I gradually populate that document with data?

---

<div class="post-metadata">

**Author:** ![ThomasB](https://avatars.discourse-cdn.com/v4/letter/t/f07891/32.png) [@ThomasB](https://discuss.elastic.co/u/ThomasB)\
**Post date:** [December 7, 2015, 9:49pm UTC](https://discuss.elastic.co/t/how-to-progressively-update-my-document/36575/2 "2015-12-07T21:49:28Z")

</div>

Hello  
The default update action in logstash (and in elasticsearch) replaces the document

You want to use the upsert option:

> **[Logstash 1.5.4 and 1.4.5 released](https://www.elastic.co/blog/logstash-1-5-4-and-1-4-5-released)**
>
> We are announcing the release of logstash 1.5.4 and 1.4.5 which fixes important security issues.

  
[https://www.elastic.co/guide/en/elasticsearch/guide/current/partial-updates.html](https://www.elastic.co/guide/en/elasticsearch/guide/current/partial-updates.html)

If you want to only have completed orders in your elasticsearch, you can also use the aggregate filter in logstash. This should do the same trick, without writing to the database 7 times per order  
[https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html](https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html)

I hope this helps

---

<div class="post-metadata">

**Author:** ![jjdepaul](https://avatars.discourse-cdn.com/v4/letter/j/e0b2c6/32.png) [@jjdepaul](https://discuss.elastic.co/u/jjdepaul)\
**Post date:** [December 7, 2015, 10:02pm UTC](https://discuss.elastic.co/t/how-to-progressively-update-my-document/36575/3 "2015-12-07T22:02:57Z")

</div>

This is what I have setup, but still no dice:

> ```
> elasticsearch {
> hosts => localhost
> index => "orders_alt"
> action => "update"
> user => "es_admin"
> password => "pa$$word"
> document_id => "%{env}_%{orderNumber}"
> doc_as_upsert => true
> }
> 
> ```

That's what the examples showed - why doesn't that preserve the data following incremental updates to the same document?

---

<div class="post-metadata">

**Author:** ![Architha](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/architha/32/92477_2.png) [@Architha](https://discuss.elastic.co/u/Architha)\
**Post date:** [February 15, 2016, 10:16am UTC](https://discuss.elastic.co/t/how-to-progressively-update-my-document/36575/4 "2016-02-15T10:16:44Z")

</div>

Hi  
I am facing a similar situation , need to update the documents based on the document ID, but its replacing the existing document with the new document with same doc\_id.  
Please help! Thanks in Advance.

---

<div class="post-metadata">

**Author:** ![jjdepaul](https://avatars.discourse-cdn.com/v4/letter/j/e0b2c6/32.png) [@jjdepaul](https://discuss.elastic.co/u/jjdepaul)\
**Post date:** [February 15, 2016, 2:57pm UTC](https://discuss.elastic.co/t/how-to-progressively-update-my-document/36575/5 "2016-02-15T14:57:01Z")

</div>

Check your "action", it should be " **upsert**", not "update".

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:11am UTC](https://discuss.elastic.co/t/how-to-progressively-update-my-document/36575/6 "2017-07-06T05:11:26Z")

</div>