org.elasticsearch.hadoop.rest.EsHadoopRemoteException: illegal_argument_exception: value for key [X-Opaque-Id] already present

[Akhil_parmar]

I am working on ETL job where target to load data to elasticsearch, I am facing an error when trying to index document
org.elasticsearch.hadoop.rest.EsHadoopRemoteException: illegal_argument_exception: value for key [X-Opaque-Id] already present

Versions:
ES: 7.6.1
ES spark SDK: 7.17.0
java: 8
spark: 3.1.0

Can someone please help if I am missing any config

[Keith_Massey]

Hi @Akhil_parmar. Can you paste the stack trace from your elasticsearch logs? Also, what request were you making at the time of the error?

[Akhil_parmar]

I was making an index request,
StackTrace:
org.elasticsearch.hadoop.rest.EsHadoopRemoteException: illegal_argument_exception: value for key [X-Opaque-Id] already present
{"index":{"_id"::123"}}
{"id":"123", "name":"abc"}

We are using open source version of Elasticsearch hosted on private cloud.

I could see that X-Opaque-id header was added in 7.16.0 version.
To resolve the issue I downgraded the ES spark SDK

<!-- https://mvnrepository.com/artifact/org.elasticsearch/elasticsearch-spark-30 -->
<dependency>
    <groupId>org.elasticsearch</groupId>
    <artifactId>elasticsearch-spark-30_2.12</artifactId>
    <version>7.15.0</version>
    <scope>provided</scope>
</dependency>