# Elasticsearch-hadoop not discovering all fields in the index

**URL:** <https://discuss.elastic.co/t/elasticsearch-hadoop-not-discovering-all-fields-in-the-index/86575>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [May 21, 2017, 3:29pm UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-not-discovering-all-fields-in-the-index/86575 "2017-05-21T15:29:10Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Dustin\_Decker](https://avatars.discourse-cdn.com/v4/letter/d/e495f1/32.png) [@Dustin\_Decker](https://discuss.elastic.co/u/Dustin_Decker)\
**Post date:** [May 21, 2017, 3:29pm UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-not-discovering-all-fields-in-the-index/86575/1 "2017-05-21T15:29:11Z")

</div>

The index i'm querying has over 400 fields, and each document contains a subset of these fields.

When performing a simple count query on a src\_ip field, I see get this exception:

```
Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot resolve '`src_ip`' given input columns: [@timestamp, event_timestamp, geoip, type]; line 1 pos 7;

```

It only recognizes 4 out of the \>400 fields.

I messed with `es.read.field.include` but found it only works within the input columns in the list above.

How can I perform queries against all of my fields?

---

<div class="post-metadata">

**Author:** ![Dustin\_Decker](https://avatars.discourse-cdn.com/v4/letter/d/e495f1/32.png) [@Dustin\_Decker](https://discuss.elastic.co/u/Dustin_Decker)\
**Post date:** [May 21, 2017, 7:03pm UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-not-discovering-all-fields-in-the-index/86575/2 "2017-05-21T19:03:52Z")

</div>

I found my error.

When declaring the dataframe, you need to specify `<index-name>/<document-type>`. I was only specifying the index name.

This is what worked:  
val df = sqlContext.read.format(esFormat).options(elasticOptions).load("data-2017.05.21/sometype")

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 18, 2017, 7:04pm UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-not-discovering-all-fields-in-the-index/86575/3 "2017-06-18T19:04:31Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.