# Performance Challenge
**URL:** https://discuss.elastic.co/t/performance-challenge/80509
**Category:** Elasticsearch
**Tags:** es-hadoop
**Created:** [March 29, 2017, 3:27pm UTC](https://discuss.elastic.co/t/performance-challenge/80509 "2017-03-29T15:27:41Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@krajaram](https://discuss.elastic.co/u/krajaram)
#### Post date: [March 29, 2017, 3:27pm UTC](https://discuss.elastic.co/t/performance-challenge/80509/1 "2017-03-29T15:27:41Z")
I have a small 3 node Elasticsearch cluster (each VM 8 Cores, 28 Gb RAM, 56 GB SSD). I am using Spark Streaming (Dstreams) to stream data from MQTT broker into ES. The index has 50M documents. When I use spark sql and create a dataframe from this index and do a count on the dataframe it takes 1 hr to return the result. If I write the dataframe as parquet file in HDFS the count returns in \<1 sec. So I am suspecting there is some issue with the way I am using Elasticsearch for Hadoop. Not sure how to resolve this issue.
---
### Author:  [@jkuang](https://discuss.elastic.co/u/jkuang)
#### Post date: [March 29, 2017, 6:10pm UTC](https://discuss.elastic.co/t/performance-challenge/80509/2 "2017-03-29T18:10:07Z")
How does it take to run the count directly from ES? This test will show whether it's the connection between ES and Spark Streaming (Dstreams) that is slow.
---
### Author:  [@krajaram](https://discuss.elastic.co/u/krajaram)
#### Post date: [March 29, 2017, 6:21pm UTC](https://discuss.elastic.co/t/performance-challenge/80509/3 "2017-03-29T18:21:16Z")
It takes less than a second. There is no issue while writing the index using Dstreams; it is only when trying to read the index it is very slow. I even installed Spark on one of the ES nodes and ran the same query; still the response was very slow.
---
### Author:  [@jkuang](https://discuss.elastic.co/u/jkuang)
#### Post date: [March 29, 2017, 8:21pm UTC](https://discuss.elastic.co/t/performance-challenge/80509/4 "2017-03-29T20:21:43Z")
Interesting, so reading / running the count directly from ES is fine. However, when you read from the Spark console it takes a long time. It sounds like to me it might be a Spark pagination or network problem. Does the slow read only happen to one index or all your indices?
---
### Author:  [@krajaram](https://discuss.elastic.co/u/krajaram)
#### Post date: [March 29, 2017, 9:23pm UTC](https://discuss.elastic.co/t/performance-challenge/80509/5 "2017-03-29T21:23:26Z")
I have only two indices in the cluster. The other index is small with about 2 M documents. The counts on that is also slow; it takes about 23 secs
---
### Author:  [@krajaram](https://discuss.elastic.co/u/krajaram)
#### Post date: [March 31, 2017, 2:49pm UTC](https://discuss.elastic.co/t/performance-challenge/80509/6 "2017-03-31T14:49:33Z")
I discussed with Elastic yesterday and learnt that count(\*) or any other form of aggregation is not pushed down by ES for Spark. They recommended to use the Java REST API within Spark code for aggregations.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 28, 2017, 2:49pm UTC](https://discuss.elastic.co/t/performance-challenge/80509/7 "2017-04-28T14:49:44Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.