# Spark sql not reading all the columns from index

**URL:** <https://discuss.elastic.co/t/spark-sql-not-reading-all-the-columns-from-index/215518>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [January 17, 2020, 8:25pm UTC](https://discuss.elastic.co/t/spark-sql-not-reading-all-the-columns-from-index/215518 "2020-01-17T20:25:06Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![raghu2](https://avatars.discourse-cdn.com/v4/letter/r/43a26b/32.png) [@raghu2](https://discuss.elastic.co/u/raghu2)\
**Post date:** [January 17, 2020, 8:25pm UTC](https://discuss.elastic.co/t/spark-sql-not-reading-all-the-columns-from-index/215518/1 "2020-01-17T20:25:06Z")

</div>

I am using pyspark 1.6.1 using elasticsearch-spark-13\_2.10-7.5.1.jar to read data from ES 5.6.8 running on AWS Es service.  
I am able to use "es.read.field.include" to extract only the columns we need or also register the index as temp table and select only the columns we need.  
The source index mapping is not auto updated and any columns that are not in the index mapping are not available to extract from spark.  
How do we read all the columns from ES using spark.  
I tried to pass es.query = {"query": {"match\_all": {}} }" as well but still it uses the index mapping for the schema.  
Is there a way i can extract the whole index and create a data frame on that data.

---

<div class="post-metadata">

**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\
**Post date:** [January 28, 2020, 9:44pm UTC](https://discuss.elastic.co/t/spark-sql-not-reading-all-the-columns-from-index/215518/2 "2020-01-28T21:44:17Z")

</div>

Can you share the mapping from your Elasticsearch index?

> The source index mapping is not auto updated and any columns that are not in the index mapping are not available to extract from spark.

I'm a little confused about this, can you elaborate how you have your index set up? ES-Hadoop depends on up to date mappings on Elasticsearch in order to determine what data type each field is.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 25, 2020, 9:44pm UTC](https://discuss.elastic.co/t/spark-sql-not-reading-all-the-columns-from-index/215518/3 "2020-02-25T21:44:21Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 25, 2020, 9:44pm UTC](https://discuss.elastic.co/t/spark-sql-not-reading-all-the-columns-from-index/215518/4 "2020-02-25T21:44:22Z")

</div>



---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 1, 2020, 11:18pm UTC](https://discuss.elastic.co/t/spark-sql-not-reading-all-the-columns-from-index/215518/5 "2020-06-01T23:18:31Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.