Logstash is overwriting output elasticsearch

Hi,
Is it possible to add a field in elasticsearch output only if the document in the index does not contain that field?

I'm trying to fetch data from a API (using the input http_poller) and storing the data in a elasticsearch index (using output elasticsearch). If the index does not contains a specific field i want to add it (with a default value). I need to check its existence because I have action 'update' and doc_as_upsert true.

Currently I'm using the "add_field" in the http_poller but at every schedule it overwrite the current value of the field stored in the index.

Current:

input {
  http_poller {
    urls => {
      test1 => "http://localhost:9000/test"
    }
    request_timeout => 60
    schedule => { cron => "* * * * *"}
    codec => "json"
    metadata_target => "http_poller_metadata"
	add_field => { "status" => "active"}
  }
}

output {
	elasticsearch {
		hosts => ["localhost:9200"]
		index => "test-2020.08.31"
		document_id => "%{[uid]}"
		action => 'update'
		doc_as_upsert => true
	}
}

When the value of "status" change after a schedule (eg. from "active" to "stopped") at the next schedule it will overwrite the current value (from "stopped" to "active")

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.