I am having a problem with AWS Elasticsearch 5.1.1.
Firstly, I setup manual cluster with 2 nodes with Elasticsearch 5.1.1 on local machines and I used Elasticseach client 5.1.1 and Spark connector and everything was working fine. But when I switched to AWS Elasticsearch 5.1.1 and tried the same thing, it starts giving me errors like Cluster not accessible. e.t.c I found that it's because AWS Elasticsearch does not support transport port. So, I am limited to use only HTTP port 80. When I try rest API it works fine but it's slow because I am ingesting massive data.
Secondly, I tried to ingest directly from Spark using DataFrames it again gave me the error.
Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'