# "doc\_as\_upsert" overwrites logs instead of appending the logs

**URL:** <https://discuss.elastic.co/t/doc-as-upsert-overwrites-logs-instead-of-appending-the-logs/306898>\
**Category:** Logstash\
**Created:** [June 10, 2022, 2:42pm UTC](https://discuss.elastic.co/t/doc-as-upsert-overwrites-logs-instead-of-appending-the-logs/306898 "2022-06-10T14:42:20Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Marc\_Jacques](https://avatars.discourse-cdn.com/v4/letter/m/46a35a/32.png) [@Marc\_Jacques](https://discuss.elastic.co/u/Marc_Jacques)\
**Post date:** [June 10, 2022, 2:42pm UTC](https://discuss.elastic.co/t/doc-as-upsert-overwrites-logs-instead-of-appending-the-logs/306898/1 "2022-06-10T14:42:20Z")

</div>

Hello Elastic,

I have a filebeat agent that harvests logs. It sends the logs to Logstash on the same server, which sends it after to Elastic Cloud, the field "message" has the value of the log on Elastic cloud.

After a document has been created, if there are updates, I would like to append them to the existing logs in the same document, in the field "message". But unfortunately, the update overwrites the previous log in "message"

If the log is updated, filebeat sends to logstash only the updated message, and not all of it to logstash, which seems the normal course.

Tried lot of combination, but I can't get it to work, everytime the field message is overwritten, by the updated message in the log  
If you could please help me figure it out please?

My logstash conf

```auto
input {
  beats {
    port => 5044
    add_field => {
      "[@metadata][target_index]" => "index-[name]-log-%{+YYYY-MM-dd}"
      "[@metadata][target_document_id]" => "%{[log][file][path]}"
    }
  }
}

output {
  elasticsearch {
   action => "update"
   hosts => ["https://[host].northeurope.azure.elastic-cloud.com:*"]
   user => "elastic"
   password => "***"
   proxy => "***"
   index => "%{[@metadata][target_index]}"
   doc_as_upsert => true
   document_id => "%{[@metadata][target_document_id]}"
  }
}

```

version 7.17.2 Elasticsearch  
version 8.1.3 Logstash  
version 8.1.3 Filebeat

Thanks

---

<div class="post-metadata">

**Author:** ![Tomo\_M](https://avatars.discourse-cdn.com/v4/letter/t/848f3c/32.png) [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)\
**Post date:** [June 11, 2022, 4:19am UTC](https://discuss.elastic.co/t/doc-as-upsert-overwrites-logs-instead-of-appending-the-logs/306898/2 "2022-06-11T04:19:12Z")

</div>

[doc\_as\_upsert](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update.html#doc_as_upsert) have effect only when there is no previous document and newly indexing a document with `update` mode.  
without doc\_as\_upsert, a separate `upsert` field is used to index a new document.  
with doc\_as\_upsert, a `doc` field (which is commonly used to update existing document) is used to index a new document.

That is not a function to append a new value to an existing field. I have no good idea to achieve it. You may need an elastic filter plugin to get existing document and a ruby script filter plugin to merge new value to the existing array.

Why don't you index every new message as a new document and aggregate them if necessary? I recommend such way.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 9, 2022, 4:19am UTC](https://discuss.elastic.co/t/doc-as-upsert-overwrites-logs-instead-of-appending-the-logs/306898/3 "2022-07-09T04:19:23Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.