Hi,
I am trying to read from elasticsearch and create a spark dataframe. I am getting following error.
ERROR 2018-01-12 17:59:00,603 Executor task launch worker-1 DefaultWriterContainer:96 - Aborting task.
org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: failed to parse [1.6h], fractional time values are not supported
{"query":{"bool":{"must":[{"match_all":{"boost":1.0}}],"filter":[{"terms":{"user_id":["106616"],"boost":1.0}}],"disable_coord":false,"adjust_pure_negative":true,"boost":1.0}}}
at org.elasticsearch.hadoop.rest.RestClient.checkResponse(RestClient.java:505)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:463)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:445)
at org.elasticsearch.hadoop.rest.RestRepository.scroll(RestRepository.java:365)
at org.elasticsearch.hadoop.rest.ScrollQuery.hasNext(ScrollQuery.java:92)
at org.elasticsearch.spark.rdd.AbstractEsRDDIterator.hasNext(AbstractEsRDDIterator.scala:61)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:350)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at org.apache.spark.sql.execution.datasources.DefaultWriterContainer.writeRows(WriterContainer.scala:240)
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1$$anonfun$apply$mcV$sp$3.apply(InsertIntoHadoopFsRelation.scala:150)
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1$$anonfun$apply$mcV$sp$3.apply(InsertIntoHadoopFsRelation.scala:150)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:88)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
INFO 2018-01-12 17:59:00,605 Executor task launch worker-0 InternalParquetRecordWriter:151 - Flushing mem columnStore to file. allocated memory: 0
INFO 2018-01-12 17:59:00,605 Executor task launch worker-1 InternalParquetRecordWriter:151 - Flushing mem columnStore to file. allocated memory: 0
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
ERROR 2018-01-12 17:59:00,662 Executor task launch worker-1 DefaultWriterContainer:75 - Task attempt attempt_201801121758_0000_m_000001_0 aborted.
ERROR 2018-01-12 17:59:00,662 Executor task launch worker-0 DefaultWriterContainer:75 - Task attempt attempt_201801121758_0000_m_000000_0 aborted.
ERROR 2018-01-12 17:59:00,667 Executor task launch worker-1 Executor:96 - Exception in task 1.0 in stage 0.0 (TID 1)
org.apache.spark.SparkException: Task failed while writing rows.
at org.apache.spark.sql.execution.datasources.DefaultWriterContainer.writeRows(WriterContainer.scala:250)
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1$$anonfun$apply$mcV$sp$3.apply(InsertIntoHadoopFsRelation.scala:150)
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1$$anonfun$apply$mcV$sp$3.apply(InsertIntoHadoopFsRelation.scala:150)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:88)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
We are using
elasticsearch 5.6.4.
Spark 1.6.0
Please suggest to fix this issue.