# Query filter not working with SparkSql **URL:** \ **Category:** Elasticsearch\ **Tags:** es-hadoop\ **Created:** [January 25, 2017, 1:50am UTC](https://discuss.elastic.co/t/query-filter-not-working-with-sparksql/72719 "2017-01-25T01:50:47Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![zpp](https://avatars.discourse-cdn.com/v4/letter/z/54ee81/32.png) [@zpp](https://discuss.elastic.co/u/zpp)\ **Post date:** [January 25, 2017, 1:50am UTC](https://discuss.elastic.co/t/query-filter-not-working-with-sparksql/72719/1 "2017-01-25T01:50:47Z")
I'm trying to pull data from Elasticsearch using below two commands, both returned data with the same record count, however, sql\_rdd returned all fields in elasticsearch, while es\_rdd only returned timestamp, host and message fields as specified in the query filter. Query strings are the the same. Is the way I use SparkSql correct? How to make the filter work for SparkSql? Thanks a lot! sql\_rdd = sqlContext.read.format("org.elasticsearch.spark.sql").option("es.nodes", "serverA").option("es.query", "{"fields": ["@timestamp", "host", "message"], "query": { "filtered": { "query": {"match\_all": {}}, "filter": {"range": { "@timestamp": { "gte": 1485050400000, "lt": 1485050430000} } } } } }").load("logstash-2017.01.22") es\_rdd = sc.newAPIHadoopRDD( inputFormatClass="org.elasticsearch.hadoop.mr.EsInputFormat", keyClass="org.apache.hadoop.io.NullWritable", valueClass="org.elasticsearch.hadoop.mr.LinkedMapWritable", conf = {"es.nodes": "serverA", "es.resource": "logstash-2017.01.22", "es.query": "{"fields": ["@timestamp", "host", "message"], "query": { "filtered": { "query": {"match\_all": {}}, "filter": {"range": { "@timestamp": { "gte": 1485050400000, "lt": 1485050430000} } } } } }"}) ---
**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\ **Post date:** [January 25, 2017, 9:00pm UTC](https://discuss.elastic.co/t/query-filter-not-working-with-sparksql/72719/2 "2017-01-25T21:00:01Z")
@zpp Could you include the versions of the technologies used? ---
**Author:** ![zpp](https://avatars.discourse-cdn.com/v4/letter/z/54ee81/32.png) [@zpp](https://discuss.elastic.co/u/zpp)\ **Post date:** [January 26, 2017, 1:48am UTC](https://discuss.elastic.co/t/query-filter-not-working-with-sparksql/72719/3 "2017-01-26T01:48:02Z")
I'm using elasticsearch 2.4.0, elasticsearch-hadoop 2.4.0, and Spark 1.6.2 ---
**Author:** ![zpp](https://avatars.discourse-cdn.com/v4/letter/z/54ee81/32.png) [@zpp](https://discuss.elastic.co/u/zpp)\ **Post date:** [January 31, 2017, 7:48am UTC](https://discuss.elastic.co/t/query-filter-not-working-with-sparksql/72719/4 "2017-01-31T07:48:39Z")
Anyone can help? thanks a lot! ---
**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\ **Post date:** [January 31, 2017, 4:24pm UTC](https://discuss.elastic.co/t/query-filter-not-working-with-sparksql/72719/5 "2017-01-31T16:24:47Z")
@zpp Use of anything other than the `"query"` element in the `es.query` configuration is not officially supported. In 5.0, the logic for handling the `es.query` option was unified across all integrations. In 5.2.0 there will be an officially supported avenue for specifying the desired source fields from the request. ---
**Author:** ![zpp](https://avatars.discourse-cdn.com/v4/letter/z/54ee81/32.png) [@zpp](https://discuss.elastic.co/u/zpp)\ **Post date:** [February 1, 2017, 5:49am UTC](https://discuss.elastic.co/t/query-filter-not-working-with-sparksql/72719/6 "2017-02-01T05:49:33Z")
Thanks a lot for the information, James. I assume you're referring to ElasticSearch-Hadoop 5.2.0, which was just released on Jan 31. I did a quick testing on this version, without upgrading the elasticsearch cluster (not sure this is ok, but there are no errors). Using the same commands above, however, both returned all fields rather than the seletcted ones, even for newAPIHadoopRDD command, which was working with version 2.4. ---
**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\ **Post date:** [February 2, 2017, 7:44pm UTC](https://discuss.elastic.co/t/query-filter-not-working-with-sparksql/72719/7 "2017-02-02T19:44:59Z")
@zpp You can enable the field selections by using the `es.read.source.filter` as detailed in [the docs for 5.2.0](https://www.elastic.co/guide/en/elasticsearch/hadoop/current/configuration.html#configuration-options-index). > [@zpp](#): > > not sure this is ok, but there are no errors ES-Hadoop should be backwards compatible with ES 2.4.0, so no sweat 🙂 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 2, 2017, 7:45pm UTC](https://discuss.elastic.co/t/query-filter-not-working-with-sparksql/72719/8 "2017-03-02T19:45:36Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.