Invalid major version 2.5.0

[spearsear]

I have two OpenSearch 2.5 clusters running in 2 AWS accounts (environments), dev and stg.
I use elasticsearch-spark-30_2.12-7.15.2.jar to write data to an index in OpenSearch.

dev works fine, but stg throws error below:

An error occurred while calling o624.save.: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'

Then look further I spot this error:

org.elasticsearch.hadoop.EsHadoopIllegalStateException: Invalid major version [2.5.0]. Version is lower than minimum required version [6.x]

My dev and stg environments are created using cloud formation, they should be exactly identical. Both environments have OpenSearch 2.5. Why I could write to dev but not stg?

Also I think OpenSearch is forked from Elasticsearch 7.X so it's already higher then 6.X. Why the error message complains on 2.5.0 which is an OpenSearch version, not a elasticsearch version?

I tried elasticsearch-spark-30_2.12-8.9.0.jar, still have the same problem.

My elasticsearch-spark code to write to opensearch is, which works for dev:

df.write.format(
            'org.elasticsearch.spark.sql'
        ).option(
            'es.nodes.wan.only', 'true'
        ).option(
            'es.nodes', 'my_domain_url'
        ).option(
            'es.port', '443'
        ).option(
            'es.resource', 'my_index',
        ).option(
            "es.write.operation", "upsert"
        ).option(
            "es.mapping.id", "doc_id"
        ).option(
            "es.batch.size.entries", "100"
        ).option(
            "es.batch.write.retry.wait", "60s"
        ).option(
            "es.batch.write.retry.count", 6
        ).mode(
            "append"
        ).save()

Any one can shed some insight? I know this question has to do with OpenSearch, but since I am using elasticsearch-spark.jar, this question is valid on this board.

Thanks for your help!

[system]

OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns )

[TimV (Tim Vernum)]

Why the error message complains on 2.5.0 which is an OpenSearch version, not a elasticsearch version?

Because elasticsearch-spark is only designed to work with Elasticsearch. It asks the cluster for its version, and gets told that it is "2.5" which is lower than 6

If you want help connecting from Spark to OpenSearch then you probably need to ask the OpenSearch project - elasticsearch-spark isn't intended to work for that.

[spearsear]

Thanks for answering. But why elasticsearch-spark works in one of my environment?

[spearsear]

Got it now, I just need to enable compatibility mode in OpenSearch, I perhaps did it in my dev domain but not doing it in stg domain:

PUT /_cluster/settings
{
  "persistent" : {
    "compatibility.override_main_response_version" : true
  }
}