Hello, I have two pipelines modifying different fields from the same documents and each time one pipeline runs, it overwrites the information of the other pipeline's fields.
I have a large (150K) product database in ElasticSearch. The source is my transactional mysql database, which ingests new and updated products into ElasticSearch through AppSearch and Logstash.
I have two pipelines with jdbc as input and appsearch as output plugin. This queries my database and generates POST requests to AppSearch, which internally modifies the underlying ElasticSearch.
The first one runs every 5 minutes, inserts new records and should update almost every field of the existing documents. I need this to be scheduled as frequent as possible.
The other is much heavier, runs every hour and should update just 1 or 2 fields of all the existing documents.
When the heavy pipeline finishes writing this 1 or 2 fields, 5 minutes later the other pipeline runs, and despite the input event doesn't include those 1 or 2 special fields, it blanks them anyway.
Since AppSearch is very limited in advanced query operations, I can't make one index (or Engine in AppSearch) for each pipeline and join them when searching. I need all the information in the same index.
I also thought as workaround to use some filter to query the elasticsearch/appsearch document being modified in order to get the rest of the fields' values and then send the full event to appsearch. This is a very inefficient solution, if even possible.
How can have this two pipelines modify only specific fields form the same documents without overwriting each other?
Thank you in advance