# Update existing and insert new fields on the same index in elastic search using Logstash config

**URL:** <https://discuss.elastic.co/t/update-existing-and-insert-new-fields-on-the-same-index-in-elastic-search-using-logstash-config/97500>\
**Category:** Logstash\
**Created:** [August 18, 2017, 12:17am UTC](https://discuss.elastic.co/t/update-existing-and-insert-new-fields-on-the-same-index-in-elastic-search-using-logstash-config/97500 "2017-08-18T00:17:58Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![vreddy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vreddy/32/20776_2.png) [@vreddy](https://discuss.elastic.co/u/vreddy)\
**Post date:** [August 18, 2017, 12:17am UTC](https://discuss.elastic.co/t/update-existing-and-insert-new-fields-on-the-same-index-in-elastic-search-using-logstash-config/97500/1 "2017-08-18T00:17:58Z")

</div>

Hi,

I have scenario where I need to update the document on the elastic search from two different sources, I am creating same document id while processing 2 sources. Lets say at first document has A, B, C fields from first source. Now, second source updates A, B and adds a new field D.  
I am expecting the document to be A, B, C, D with updated fields but this is not happening with following output plugin config. Any help is appreciated.

- 

```
 input{
     file {
      path => "/xyz/**/*.txt"
      start_position => "beginning"
      # ignore_older => 0
      type => "legacy"
      codec => multiline {
       pattern => "^=== Executing command: .*===" 
        negate => true
        what => "previous"
        max_lines => 20000
        max_bytes => "100 MiB"
      }
    }
  }
  filter {
  .
  .
  }
  output {
  elasticsearch {
        hosts => ["elasticsearch:9200"]
        index => "cluster-details-ts-%{+YYYY-MM-dd}"
        document_id => "%{fingerprint}"
        retry_initial_interval => 10
        retry_max_interval => 300
        retry_on_conflict => 25
        action => "update"
        doc_as_upsert => "true"
     }
  } 

```

sample input is:  
=== Executing command: echo system.hostname: ControllerHHX8RPSFEL ===  
system.hostname: ControllerHHX8RPSFEL  
=== Command succeeded: echo system.hostname: ControllerHHX8RPSFEL ===

=== Executing command: uname -a ===  
Linux ControllerHHX8RPSFEL 3.2.0-58-generic #88-Ubuntu SMP Tue Dec 3 17:37:58 UTC 2013 x86\_64 x86\_64 x86\_64 GNU/Linux  
=== Command succeeded: uname -a ===

From first command output I am extracting hostname and from second one hostname, OS, etc

After processing two command outputs document should have hostname, OS, etc with updated fields.. but its not happening.

* * *

---

<div class="post-metadata">

**Author:** ![thiago](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thiago/32/32096_2.png) [@thiago](https://discuss.elastic.co/u/thiago)\
**Post date:** [August 18, 2017, 12:36am UTC](https://discuss.elastic.co/t/update-existing-and-insert-new-fields-on-the-same-index-in-elastic-search-using-logstash-config/97500/2 "2017-08-18T00:36:45Z")

</div>

What is happening? Documents are being overwritten? Of many separate documents are being written?

Also, your multiline pattern does not seems right to me. I would expect it to be like:

```auto
^=== Executing command: .+ ===$

```

---

<div class="post-metadata">

**Author:** ![vreddy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vreddy/32/20776_2.png) [@vreddy](https://discuss.elastic.co/u/vreddy)\
**Post date:** [August 18, 2017, 6:29am UTC](https://discuss.elastic.co/t/update-existing-and-insert-new-fields-on-the-same-index-in-elastic-search-using-logstash-config/97500/3 "2017-08-18T06:29:17Z")

</div>

Yes, you are correct my pattern is ^=== Executing command: .\* ===$

Documents are getting overwritten instead of updating or adding new fields to same docId document.

---

<div class="post-metadata">

**Author:** ![vreddy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vreddy/32/20776_2.png) [@vreddy](https://discuss.elastic.co/u/vreddy)\
**Post date:** [August 18, 2017, 5:26pm UTC](https://discuss.elastic.co/t/update-existing-and-insert-new-fields-on-the-same-index-in-elastic-search-using-logstash-config/97500/4 "2017-08-18T17:26:47Z")

</div>

@thiago, Could you please help me find out the issue here. Thanks

---

<div class="post-metadata">

**Author:** ![vreddy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vreddy/32/20776_2.png) [@vreddy](https://discuss.elastic.co/u/vreddy)\
**Post date:** [August 18, 2017, 6:22pm UTC](https://discuss.elastic.co/t/update-existing-and-insert-new-fields-on-the-same-index-in-elastic-search-using-logstash-config/97500/5 "2017-08-18T18:22:33Z")

</div>

Hey sorry, I had a typo in fingerprint generation which I use as docId.  
It is working as expected.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 15, 2017, 6:22pm UTC](https://discuss.elastic.co/t/update-existing-and-insert-new-fields-on-the-same-index-in-elastic-search-using-logstash-config/97500/6 "2017-09-15T18:22:56Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.