Hi Team
Am trying to fetch data from elastic server into spark 2.4 environment using "org.elasticsearch.spark.sql" connector, but even though i got all the configs right, there is no data received on the spark engine
Using the below config to connect to Elastic from spark
On analyzing with fiddler found that the request created had a search_type=scan param in the query string, which i guess is no longer supported
http://NodeURL/index/type/_search?search_type=scan&scroll=5m&size=50&preference=_shards%3A3%3B_local
On opening this on the browser didn't receive any data, but the same query without the search_type param worked as expected.
Am wondering when the above param is no longer supported, why is it being appended in the request url by the elasticsearch-spark connector
Can someone please help me figure out a proper solution
Spark - 2.4.0
scala - 2.11
elasticsearch-spark-20_2.11 (6.6.0)
Thanks
John