I am using pyspark 1.6.1 using elasticsearch-spark-13_2.10-7.5.1.jar to read data from ES 5.6.8 running on AWS Es service.
I am able to use "es.read.field.include" to extract only the columns we need or also register the index as temp table and select only the columns we need.
The source index mapping is not auto updated and any columns that are not in the index mapping are not available to extract from spark.
How do we read all the columns from ES using spark.
I tried to pass es.query = {"query": {"match_all": {}} }" as well but still it uses the index mapping for the schema.
Is there a way i can extract the whole index and create a data frame on that data.
Can you share the mapping from your Elasticsearch index?
The source index mapping is not auto updated and any columns that are not in the index mapping are not available to extract from spark.
I'm a little confused about this, can you elaborate how you have your index set up? ES-Hadoop depends on up to date mappings on Elasticsearch in order to determine what data type each field is.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.