# Spark SQL and many long field names **URL:** \ **Category:** Elasticsearch\ **Tags:** es-hadoop\ **Created:** [February 24, 2017, 11:07pm UTC](https://discuss.elastic.co/t/spark-sql-and-many-long-field-names/76458 "2017-02-24T23:07:58Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![cj\_usermind](https://avatars.discourse-cdn.com/v4/letter/c/a4c791/32.png) [@cj\_usermind](https://discuss.elastic.co/u/cj_usermind)\ **Post date:** [February 24, 2017, 11:07pm UTC](https://discuss.elastic.co/t/spark-sql-and-many-long-field-names/76458/1 "2017-02-24T23:07:59Z")
I have an index with many long field names. When using Spark SQL via Spark Thrift Server, if I create a temporary view and then query it, I get an authorization error, e.g., CREATE GLOBAL TEMPORARY VIEW view1 USING org.elasticsearch.spark.sql OPTIONS (resource 'es-index'); SELECT \* from view1 limit 1; failed; server[[docker-prod.west.usermind.com:8200](http://docker-prod.west.usermind.com:8200)] returned [401|Unauthorized:] at org.elasticsearch.hadoop.rest.RestClient.checkResponse(RestClient.java:488) at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:446) at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:436) at org.elasticsearch.hadoop.rest.RestRepository.scroll(RestRepository.java:363) at org.elasticsearch.hadoop.rest.ScrollQuery.hasNext(ScrollQuery.java:92) at org.elasticsearch.spark.rdd.AbstractEsRDDIterator.hasNext(AbstractEsRDDIterator.scala:61) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408) at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:231) at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:225) at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:826) at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:826) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at org.apache.spark.scheduler.Task.run(Task.scala:99) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) I observe that the ES query is of the form es-index/\_search?sort=\_doc&scroll=5m&size=50&\_source=field1,field2,field3,...&preference=\_shards%3A0%7C\_local Through trial and error, I see that if I remove fields from the end, the query works. This points to a limit in the size of the URL being sent to ES, caused by lots of fields and long field names. My question: is there a workaround? For example, is there a parameter that can be specified to not use the query string for the underlying query to ES, but to use the query DSL? Or is there a better way to formulate the query to avoid this problem? ---
**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\ **Post date:** [March 11, 2017, 8:00pm UTC](https://discuss.elastic.co/t/spark-sql-and-many-long-field-names/76458/2 "2017-03-11T20:00:23Z")
Right now there is no option that puts the fields into the DSL body yet. I've seen [this ticket](https://github.com/elastic/elasticsearch-hadoop/issues/942) float through github recently, I'm assuming it was you who filed it (user names are different, but similar). Definitely an issue and we'll be looking into fixing. ---
**Author:** ![cj\_usermind](https://avatars.discourse-cdn.com/v4/letter/c/a4c791/32.png) [@cj\_usermind](https://discuss.elastic.co/u/cj_usermind)\ **Post date:** [March 11, 2017, 9:43pm UTC](https://discuss.elastic.co/t/spark-sql-and-many-long-field-names/76458/3 "2017-03-11T21:43:10Z")
> james.baiera [https://discuss.elastic.co/users/james.baiera](https://discuss.elastic.co/users/james.baiera) James Baiera > [https://discuss.elastic.co/users/james.baiera](https://discuss.elastic.co/users/james.baiera) > March 11 > > ## Right now there is no option that puts the fields into the DSL body yet. I've seen this ticket [https://github.com/elastic/elasticsearch-hadoop/issues/942](https://github.com/elastic/elasticsearch-hadoop/issues/942) float through github recently, I'm assuming it was you who filed it (user names are different, but similar). Definitely an issue and we'll be looking into fixing. > > Visit Topic > [https://discuss.elastic.co/t/spark-sql-and-many-long-field-names/76458/2](https://discuss.elastic.co/t/spark-sql-and-many-long-field-names/76458/2) > or reply to this email to respond. > > To unsubscribe from these emails, click here > [https://discuss.elastic.co/email/unsubscribe/6ed9a999ecfedb61d2875b8df2a3c2e2b741abcf16b8cb9ebc7acf76210585ea](https://discuss.elastic.co/email/unsubscribe/6ed9a999ecfedb61d2875b8df2a3c2e2b741abcf16b8cb9ebc7acf76210585ea) > . Yes. I filed it. We have indices in Elasticsearch that have this issue. Looking forward to the fix Chris jones ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 8, 2017, 9:43pm UTC](https://discuss.elastic.co/t/spark-sql-and-many-long-field-names/76458/4 "2017-04-08T21:43:26Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.