# Elasticsearch loses performance changing path.data running on SparkSQL **URL:** \ **Category:** Elasticsearch\ **Created:** [June 29, 2016, 4:46pm UTC](https://discuss.elastic.co/t/elasticsearch-loses-performance-changing-path-data-running-on-sparksql/54318 "2016-06-29T16:46:17Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![jackProm](https://avatars.discourse-cdn.com/v4/letter/j/9fc29f/32.png) [@jackProm](https://discuss.elastic.co/u/jackProm)\ **Post date:** [June 29, 2016, 4:46pm UTC](https://discuss.elastic.co/t/elasticsearch-loses-performance-changing-path-data-running-on-sparksql/54318/1 "2016-06-29T16:46:17Z")
Hi everyone, I'm using Elastisearch on a cluster with 4 nodes using SparkSQL. I'm trying to copy tables from the Hive-metastore to Elasticseach using saveToEs. Everything works good if I work with tables with size around 13Gb, but if I want to index a huge table with 60Gb or more something goes wrong. The problem is that many executors should write on the only folder /var/lib/elastisearch and something goes wrong. Each node of cluster has 10 hard disk, so to improve parallelism I added to path.data more folders, each per disk, so at the end I had ``` `path.data=/data1/elasticsearch,.......,/data10/elasticsearch` ``` I restarted Elastic and everything was good. I checked the new configuration with ``` `curl http://namenode:9200/_nodes/settings?pretty` ``` The problem is that with: - the default path.data the index-speed is 1Gb per minute - the new configuration speed is 1Gb per 2/2,5 minutes. So with the second configuration I can index huge table (because the process doesn't crash), but Elastic is very slow, with the default configuration Elastic is very fast, but I cannot index table bigger than 15Gb. Is there something to configure? I installed Elasticsearch 2.3.2 and I'm using the same version for maven dependecy for scala driver to use to launch Spark. Thanks ---
**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\ **Post date:** [June 30, 2016, 6:26pm UTC](https://discuss.elastic.co/t/elasticsearch-loses-performance-changing-path-data-running-on-sparksql/54318/2 "2016-06-30T18:26:55Z")
Hello! Since `path.data` is a property of Elasticsearch itself, this topic is probably better suited for the [regular Elasticsearch forum](https://discuss.elastic.co/c/elasticsearch). Unless you are saying that it's only impacted when indexing data through SparkSQL? ---
**Author:** ![jackProm](https://avatars.discourse-cdn.com/v4/letter/j/9fc29f/32.png) [@jackProm](https://discuss.elastic.co/u/jackProm)\ **Post date:** [July 1, 2016, 7:39am UTC](https://discuss.elastic.co/t/elasticsearch-loses-performance-changing-path-data-running-on-sparksql/54318/3 "2016-07-01T07:39:57Z")
Actually I tried to work with Elastic only using Spark or Hive. Do you think it's better to change my post? ---
**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\ **Post date:** [July 1, 2016, 2:18pm UTC](https://discuss.elastic.co/t/elasticsearch-loses-performance-changing-path-data-running-on-sparksql/54318/4 "2016-07-01T14:18:01Z")
Considering the only thing changed between your runs is the Elasticsearch `path.data` property (which is not a Spark or Hive property), I think you'll get faster and better feedback from the regular Elasticsearch forum. I can see if I can move the topic for you. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:38pm UTC](https://discuss.elastic.co/t/elasticsearch-loses-performance-changing-path-data-running-on-sparksql/54318/5 "2017-07-05T22:38:55Z")