Update existing and insert new fields on the same index in elastic search using Logstash config

Elastic Stack Logstash
1

Hi,

I have scenario where I need to update the document on the elastic search from two different sources, I am creating same document id while processing 2 sources. Lets say at first document has A, B, C fields from first source. Now, second source updates A, B and adds a new field D.
I am expecting the document to be A, B, C, D with updated fields but this is not happening with following output plugin config. Any help is appreciated.

  •  input{
     file {
      path => "/xyz/**/*.txt"
      start_position => "beginning"
      # ignore_older => 0
      type => "legacy"
      codec =>  multiline {
       pattern => "^=== Executing command: .*===" 
        negate => true
        what => "previous"
        max_lines => 20000
        max_bytes => "100 MiB"
      }
    }
  }
  filter {
  .
  .
  }
  output {
  elasticsearch {
        hosts => ["elasticsearch:9200"]
        index => "cluster-details-ts-%{+YYYY-MM-dd}"
        document_id => "%{fingerprint}"
        retry_initial_interval => 10
        retry_max_interval => 300
        retry_on_conflict => 25
        action => "update"
        doc_as_upsert => "true"
     }
  }

sample input is:
=== Executing command: echo system.hostname: ControllerHHX8RPSFEL ===
system.hostname: ControllerHHX8RPSFEL
=== Command succeeded: echo system.hostname: ControllerHHX8RPSFEL ===

=== Executing command: uname -a ===
Linux ControllerHHX8RPSFEL 3.2.0-58-generic #88-Ubuntu SMP Tue Dec 3 17:37:58 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
=== Command succeeded: uname -a ===

From first command output I am extracting hostname and from second one hostname, OS, etc

After processing two command outputs document should have hostname, OS, etc with updated fields.. but its not happening.

2

What is happening? Documents are being overwritten? Of many separate documents are being written?

Also, your multiline pattern does not seems right to me. I would expect it to be like:

^=== Executing command: .+ ===$
3

Yes, you are correct my pattern is ^=== Executing command: .* ===$

Documents are getting overwritten instead of updating or adding new fields to same docId document.

4

@thiago, Could you please help me find out the issue here. Thanks

5

Hey sorry, I had a typo in fingerprint generation which I use as docId.
It is working as expected.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.