Elastic - spark : deprecated search_type=scan showing up in post url from spark 2.4 and returning 0 hits

(john) #1

Hi Team

Am trying to fetch data from elastic server into spark 2.4 environment using "org.elasticsearch.spark.sql" connector, but even though i got all the configs right, there is no data received on the spark engine

Using the below config to connect to Elastic from spark

image

On analyzing with fiddler found that the request created had a search_type=scan param in the query string, which i guess is no longer supported

http://NodeURL/index/type/_search?search_type=scan&scroll=5m&size=50&preference=_shards%3A3%3B_local

On opening this on the browser didn't receive any data, but the same query without the search_type param worked as expected.

Am wondering when the above param is no longer supported, why is it being appended in the request url by the elasticsearch-spark connector

Can someone please help me figure out a proper solution

Spark - 2.4.0
scala - 2.11
elasticsearch-spark-20_2.11 (6.6.0)

Thanks
John

(David Pilato) #3

@james.baiera Do you know?

(system) closed #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.