I know this has been discussed already on previous threads, though our use-case might be a bit specific.
We have a product import process that runs on a daily basis and saves a bunch of products into an ES index (~5M items).
Every day some new products are coming in, but 99% of them are just existing products being updated.
One requirement that we have is that we need to keep track of when a product was first imported.
So far we are handling it by doing a scripted upsert: if the product already exists then don't touch the createdAt field otherwise populate it.
So this works but it turns out that the scripted upsert method is actually about 5 times slower then a normal doc as upsert method according to the metrics I get from xpack monitoring.
Unfortunately, ES does not support indexing time timestamp creation natively anymore, but is using a script to handle this the only option that we have, and if so does it mean we are stuck with a slow indexing rate while doing so? I've tried every indexing speed tuning recommendations with no success.