Optimistic concurrency control in ElasticSearch Output plugin

Elastic Stack Logstash
1

Is there a mechanism to use if_seq_no and if_primary_term in the Elasticsearch output plugin?

My use-case involves using Elasticsearch as an input (including the _seq_no, and _primary_term fields), and updating the documents using the output plugin only if they haven't been updated while the pipeline was processing the event to avoid overwrites.

I'm aware that the output plugin uses the Bulk API, and that there is an option to update each document using the if_seq_no and if_primary_term options, but I can't seem to find those options in the output plugin itself.

I'm currently using external versioning, and it works as intended, but I'd rather not resort to that kind of a solution.

Any advice would be appreciated.

2

It's not clear have you used upsert and document_id in the output.
There is a similar case where Grand Jedi Master Badger suggested the solution.

3

I'm using the "index" action to create/update documents, as it suits my needs.

I'm not facing issues in indexing the document itself, rather, I'm trying to ensure that the document won't be indexed in Elasticsearch if the sequence number and the primary term of the document in the Index has changed since the Logstash pipeline started.

The diagram below describes my use-case. The orange line is the output plugin I'm trying to modify.

image
image1201×822 83.7 KB

Note that I prioritize updates over re-ingestion.

4

Not sure if this is possible.

Maybe you would need a elasticsearch filter as the last filter before the output to get the document from elasticsearch and validate that it wasn't updated, if it was, then drop the current event.

But this also may fails as the document may be updated during the time that Logstash is sending it.