ElasticSearch Output Plugin partial update results in entire document substitution

Elastic Stack Logstash
1

Hi friends, I am having some trouble with the Elasticsearch Output Plugin, I am trying to perform a partial update on a document inside my Elastic index, I want to update certain fields only (the ones mapped in my event), but it is replacing all the document properties, and removing the undefined ones.

Eg: In Elastic I have a document with {"a" : 1. "b": 2} , if my event is {"a" : 3} , the expected result for me after update would be {"a" : 3, "b" : 2} , but its replacing it all for {"a" : 3} and removing the “b” property unfortunately, how can I manage to leave the "b" property untouched with the previous value in this case? Any help would be highly appreciated.

output {
    elasticsearch {
        hosts => ["${LOGSTASH_ELASTICSEARCH_HOST}"]
        api_key=> ["${LOGSTASH_ELASTICSEARCH_API_KEY}"]
        ssl => true
        index => "idx_envelopes"
		action => "update"
		doc_as_upsert => true
		document_id => "%{ID_Envelope}"
    }
}
2

I'm not sure if there is a shortcut for this in the output plugin but what I have done in the past is do an Elasticsearch Filter Plugin to query the document by ID and get all the fields to attach to the event. Then when it hits the output it will have all the fields available.

1 Like
3

Thanks for your response @aaron-nimocks but in my case it's a really big Text column, so I think it maybe would not be the best aproach to get the value in the elastic filter then send it back in the update, this would result in reallly big traffic/slowness IMO.

4

You need to pass in the entire document with Logstash as @aaron-nimocks mentions. Otherwise Elasticsearch assumes the update you want to apply is what is provided in the document/event with this approach.

You might be able to do it with Elasticsearch output plugin | Logstash Reference [7.15] | Elastic though.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.