Requests that lead to too many 404

Is it possible that update requests that I know will lead to 404's (because doc with that id does not exist) can destabilize an ES cluster? I'm talking about quite a lot of update requests and a big enough index (2TB)

Why I ask is because, ES is probably expected to work for the happy path i.e. document exist. This is same as the penalty you face because of CPU branch miss. We had released a change today to write to a new cluster (without any data) and old cluster (with data) & the new cluster kind of destabilized which had a ripple effect overall

I would not attribute such update requests, but maybe a rather high generic load to this. Maybe bulking requests already helps, but everything here is just wild assumptions without knowing further details. Maybe you can add some more context.

So we update using the bulk api

One thing that was super weird is that, we knew most of our bulk updates were failing because of 404, but the docs.deleted in cat indices kept on increasing like crazy. Do you happen to know anything around the calculation of docs.deleted?

To be fair, ES seemed to be doing quite well, the spikes were random, but when they happened they were in 20s/40s!

docs.deleted will increase when an update operation was successful, because the existing document is deleted and a new one is created. When a merge happens the old documents marked as deleted will be removed.