Update existing and insert new fields on the same index in elastic search using Logstash config

(grvr) #1


I have scenario where I need to update the document on the elastic search from two different sources, I am creating same document id while processing 2 sources. Lets say at first document has A, B, C fields from first source. Now, second source updates A, B and adds a new field D.
I am expecting the document to be A, B, C, D with updated fields but this is not happening with following output plugin config. Any help is appreciated.

  •  input{
         file {
          path => "/xyz/**/*.txt"
          start_position => "beginning"
          # ignore_older => 0
          type => "legacy"
          codec =>  multiline {
           pattern => "^=== Executing command: .*===" 
            negate => true
            what => "previous"
            max_lines => 20000
            max_bytes => "100 MiB"
      filter {
      output {
      elasticsearch {
            hosts => ["elasticsearch:9200"]
            index => "cluster-details-ts-%{+YYYY-MM-dd}"
            document_id => "%{fingerprint}"
            retry_initial_interval => 10
            retry_max_interval => 300
            retry_on_conflict => 25
            action => "update"
            doc_as_upsert => "true"

sample input is:
=== Executing command: echo system.hostname: ControllerHHX8RPSFEL ===
system.hostname: ControllerHHX8RPSFEL
=== Command succeeded: echo system.hostname: ControllerHHX8RPSFEL ===

=== Executing command: uname -a ===
Linux ControllerHHX8RPSFEL 3.2.0-58-generic #88-Ubuntu SMP Tue Dec 3 17:37:58 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
=== Command succeeded: uname -a ===

From first command output I am extracting hostname and from second one hostname, OS, etc

After processing two command outputs document should have hostname, OS, etc with updated fields.. but its not happening.

(Thiago Souza) #2

What is happening? Documents are being overwritten? Of many separate documents are being written?

Also, your multiline pattern does not seems right to me. I would expect it to be like:

^=== Executing command: .+ ===$

(grvr) #3

Yes, you are correct my pattern is ^=== Executing command: .* ===$

Documents are getting overwritten instead of updating or adding new fields to same docId document.

(grvr) #4

@thiago, Could you please help me find out the issue here. Thanks

(grvr) #5

Hey sorry, I had a typo in fingerprint generation which I use as docId.
It is working as expected.

(system) #6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.