# LS 6.3.2 Appending data to a field of an existing document
**URL:** https://discuss.elastic.co/t/ls-6-3-2-appending-data-to-a-field-of-an-existing-document/143878
**Category:** Logstash
**Created:** [August 10, 2018, 2:00pm UTC](https://discuss.elastic.co/t/ls-6-3-2-appending-data-to-a-field-of-an-existing-document/143878 "2018-08-10T14:00:54Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@ld\_pvl](https://discuss.elastic.co/u/ld_pvl)
#### Post date: [August 10, 2018, 2:00pm UTC](https://discuss.elastic.co/t/ls-6-3-2-appending-data-to-a-field-of-an-existing-document/143878/1 "2018-08-10T14:00:54Z")
Hi Team,
Does Logstash support appending to / extending a field's data of an existing event given a deterministic document ID?
Cheers,
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [August 10, 2018, 2:12pm UTC](https://discuss.elastic.co/t/ls-6-3-2-appending-data-to-a-field-of-an-existing-document/143878/2 "2018-08-10T14:12:21Z")
In the past I have done it by having logstash write a file that I curl into the bulk API. See [this](https://discuss.elastic.co/t/aggregate-problem/142572) thread for an example.
---
### Author:  [@ld\_pvl](https://discuss.elastic.co/u/ld_pvl)
#### Post date: [August 10, 2018, 2:29pm UTC](https://discuss.elastic.co/t/ls-6-3-2-appending-data-to-a-field-of-an-existing-document/143878/3 "2018-08-10T14:29:57Z")
Thanks Mr Badger :), it would work but it's a bit hacky.
---
### Author:  [@MikeKemmerer](https://discuss.elastic.co/u/MikeKemmerer)
#### Post date: [August 10, 2018, 3:07pm UTC](https://discuss.elastic.co/t/ls-6-3-2-appending-data-to-a-field-of-an-existing-document/143878/4 "2018-08-10T15:07:05Z")
I haven't done this, here's a suggestion:
- use the Elasticsearch filter plugin to retrieve the document by document\_id: [https://www.elastic.co/guide/en/logstash/current/plugins-filters-elasticsearch.html#plugins-filters-elasticsearch-docinfo\_fields](https://www.elastic.co/guide/en/logstash/current/plugins-filters-elasticsearch.html#plugins-filters-elasticsearch-docinfo_fields)
- in the filter, specify the field you want to append, extend via: [https://www.elastic.co/guide/en/logstash/current/plugins-filters-elasticsearch.html#plugins-filters-elasticsearch-fields](https://www.elastic.co/guide/en/logstash/current/plugins-filters-elasticsearch.html#plugins-filters-elasticsearch-fields)
- modify the field as desired
- Update the existing document by specifying the document\_id in the Elasticsearch output:
[https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-document\_id](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-document_id)
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [August 10, 2018, 3:57pm UTC](https://discuss.elastic.co/t/ls-6-3-2-appending-data-to-a-field-of-an-existing-document/143878/5 "2018-08-10T15:57:11Z")
> [@MikeKemmerer](#):
>
> I haven't done this
When I tried doing that I was [unable](https://discuss.elastic.co/t/elasticsearch-filter-giving-typeerror-no-implicit-conversion-of-nil-into-string/123791) to get it to work. I agree that it should work in principle though.
---
### Author:  [@ld\_pvl](https://discuss.elastic.co/u/ld_pvl)
#### Post date: [August 10, 2018, 4:58pm UTC](https://discuss.elastic.co/t/ls-6-3-2-appending-data-to-a-field-of-an-existing-document/143878/6 "2018-08-10T16:58:17Z")
Actually, it works!
Obviously keeping in mind that this might have performance consequences on the cluster cos now we are querying it a lot plus doing a doc update.
But here it goes:
I didn't have to use `docinfo_fields`.
```
filter {
# this applies to my scenario only but I predefined [@metadata][index]
mutate {
add_field => { "[@metadata][old_message]" => "" }
add_field => { "[@metadata][doc_id]" => "%{some_deterministic_fields_to_be_used_as_id}" } # same for this, this applies for my scenario only but the doc ID is deterministic for my case
}
elasticsearch {
hosts => "SOME_ES_HOSTS_HERE"
index => "%{[@metadata][index]}"
query => "_id:%{[@metadata][doc_id]}"
fields => { "message" => "[@metadata][old_message]" }
}
if [@metadata][old_message] != "" {
ruby {
# note that the "===NEW===" string is only for debugging purposes to see things better
code => "event.set('message', event.get('[@metadata][old_message]') + 10.chr + '===NEW===' + 10.chr + event.get('message'))"
}
}
}
```
Then for the output, in my case I did:
```
output {
elasticsearch {
hosts => "SOME_ES_HOSTS"
index => "%{[@metadata][index]}"
document_id => "%{[@metadata][doc_id]}"
doc_as_upsert => true
}
}
```
Overall, I can see the method works. However, there is one slight problem that I noticed. Because Elasticsearch persists things asynchronously, some newer events might get persisted before the old ones. So when Logstash does the Elasticsearch query, it gets returned with the newer event instead of the old event, therefore the chronological order is disrupted. This might or might not be an issue for some.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [September 7, 2018, 4:58pm UTC](https://discuss.elastic.co/t/ls-6-3-2-appending-data-to-a-field-of-an-existing-document/143878/7 "2018-09-07T16:58:18Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.