Ignoring Version Conflicts 409 with ES-Hadoop Connector

Elastic Stack Elasticsearch
1

Hi Everyone,

Is there a way to ignore version conflicts (error 409) with the ES-Hadoop Connector when using external versioning?

We are using the external version with the ES-Hadoop connector to deduplicate our data when indexing to ElasticSearch?

I also see that there is an open issue for this feature. So I guess this confirms that there is currently no feature or property to change the index behaviour as described above.

Thanks,
Mark

2

No there isn't.
This is a popular request so it might be added it. The difficult part is not ignoring the error but rather figuring it out what errors are fatal and which ones are not.

The issue with ignoring 409 is that there's no handling for it - how do you know what versions failed and which haven't; there's no logging/trapping/catching mechanism in Hadoop that works reliably (and no, commons-logging is not a solution).

These being said, since it comes up every other month, it will make it one form or the other in ES-Hadoop.