Update existing and insert new fields on the same index in elastic search using Logstash config


I have scenario where I need to update the document on the elastic search from two different sources, I am creating same document id while processing 2 sources. Lets say at first document has A, B, C fields from first source. Now, second source updates A, B and adds a new field D.
I am expecting the document to be A, B, C, D with updated fields but this is not happening with following output plugin config. Any help is appreciated.

  •  input{
         file {
          path => "/xyz/**/*.txt"
          start_position => "beginning"
          # ignore_older => 0
          type => "legacy"
          codec =>  multiline {
           pattern => "^=== Executing command: .*===" 
            negate => true
            what => "previous"
            max_lines => 20000
            max_bytes => "100 MiB"
      filter {
      output {
      elasticsearch {
            hosts => ["elasticsearch:9200"]
            index => "cluster-details-ts-%{+YYYY-MM-dd}"
            document_id => "%{fingerprint}"
            retry_initial_interval => 10
            retry_max_interval => 300
            retry_on_conflict => 25
            action => "update"
            doc_as_upsert => "true"

sample input is:
=== Executing command: echo system.hostname: ControllerHHX8RPSFEL ===
system.hostname: ControllerHHX8RPSFEL
=== Command succeeded: echo system.hostname: ControllerHHX8RPSFEL ===

=== Executing command: uname -a ===
Linux ControllerHHX8RPSFEL 3.2.0-58-generic #88-Ubuntu SMP Tue Dec 3 17:37:58 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
=== Command succeeded: uname -a ===

From first command output I am extracting hostname and from second one hostname, OS, etc

After processing two command outputs document should have hostname, OS, etc with updated fields.. but its not happening.

What is happening? Documents are being overwritten? Of many separate documents are being written?

Also, your multiline pattern does not seems right to me. I would expect it to be like:

^=== Executing command: .+ ===$

Yes, you are correct my pattern is ^=== Executing command: .* ===$

Documents are getting overwritten instead of updating or adding new fields to same docId document.

@thiago, Could you please help me find out the issue here. Thanks

Hey sorry, I had a typo in fingerprint generation which I use as docId.
It is working as expected.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.