# Weird behavior when indexing from spark **URL:** \ **Category:** Elasticsearch\ **Tags:** es-hadoop\ **Created:** [April 18, 2017, 3:25pm UTC](https://discuss.elastic.co/t/weird-behavior-when-indexing-from-spark/82738 "2017-04-18T15:25:00Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![steccami](https://avatars.discourse-cdn.com/v4/letter/s/ec9cab/32.png) [@steccami](https://discuss.elastic.co/u/steccami)\ **Post date:** [April 18, 2017, 3:25pm UTC](https://discuss.elastic.co/t/weird-behavior-when-indexing-from-spark/82738/1 "2017-04-18T15:25:00Z")
Hi all, my setup is like this: - Spark v1.6.0 (CDH 5.10) - Elastic v2.3 - es-hadoop: v 2.3.2 My spark program is something like this: `val result = sqlContext.sql("select * from tablename") result.saveToEs("index/type", cfg) val es_count=read document count from ES val hdfs_count=result.count() if (es_count!=hdfs_count) then throw Exception` Here is the weird behavior (it does not appear sistematically): - the job runs with no problems (i.e., tablename is indexed correctly and no exception is thrown); - after few seconds a few (duplicated) records are added to the index even if the job is supposed to be finished. It follows that es\_count\>hdfs\_count but the duplicated records are added after the `if (es_count!=hdfs_count)` check so no exception is fired. I thought it was something related to task re-execution but I don't see any failed task. I also tried to set `spark.task.maxFailures=1` in Spark but nothing changes. Any suggestion? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 16, 2017, 3:29pm UTC](https://discuss.elastic.co/t/weird-behavior-when-indexing-from-spark/82738/2 "2017-05-16T15:29:58Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.