# High CPU Usage on XDCR Transfer to ElasticSearch nodes

**URL:** <https://discuss.elastic.co/t/high-cpu-usage-on-xdcr-transfer-to-elasticsearch-nodes/14718>\
**Category:** Elasticsearch\
**Created:** [December 5, 2013, 1:44am UTC](https://discuss.elastic.co/t/high-cpu-usage-on-xdcr-transfer-to-elasticsearch-nodes/14718 "2013-12-05T01:44:45Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Corey\_Nolet](https://avatars.discourse-cdn.com/v4/letter/c/b19c9b/32.png) [@Corey\_Nolet](https://discuss.elastic.co/u/Corey_Nolet)\
**Post date:** [December 5, 2013, 1:44am UTC](https://discuss.elastic.co/t/high-cpu-usage-on-xdcr-transfer-to-elasticsearch-nodes/14718/1 "2013-12-05T01:44:45Z")

</div>

I've got a 6 node HDFS cluster. Each node has 8 cores and 32gb of ram. On  
the same nodes, I'm running Couchbase 2.2 and Elasticsearch 0.90.5 with the  
Couchbase transport plugin 1.1.0. I have Couchbase doing real-time  
ingest/caching from Twitter Storm and indexing the documents in  
elasticsearch for further mining. I'm using about 18gb of ram from my  
entire cluster for Couchbase. Without ElasticSearch running, Couchbase  
seems to idle between 1% and 16% CPU utilization with occasional (expected)  
spikes at around 30% to 60%. I'm okay with that.

What I'm trying to figure out, however, is why the XDCR to ElasticSearch  
seems to pump my nodes up to a constant 60% to 99% CPU utilization. I've  
tried lowering the max concurrent replications to 2 and even changing the  
checkpoint sizes and I've noticed no decrease in utilization. The  
utilization has been so high, in fact, that my Zookeeper nodes can't keep a  
lock and they start expiring important connections.

I tried pulling Elasticsearch off of my 6 nodes and putting it elsewhere  
(on a separate 4 nodes). This also didn't seem to have an impact. I read on  
a slideshow that 24-cores is recommended but that's a little absurd of a  
requirement. I'm doing about 250,000 documents every 3 minutes or so and  
ELasticsearch seems to be keeping up. The short of it is... i really just  
need the ability to do term queries of the documents and N1QL isn't nearly  
as fast with the intersecting searches as I need it to be (i'm looking for  
subsecond here).

Any ideas? I've done a ton of research and all I've seen are issues in  
verisons \<2.2 in jira that appear to have been fixed.

Thanks!

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a6920681-dd08-41f9-a781-c17bcefdcdb1%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a6920681-dd08-41f9-a781-c17bcefdcdb1%40googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Jason\_Wee](https://avatars.discourse-cdn.com/v4/letter/j/7ea924/32.png) [@Jason\_Wee](https://discuss.elastic.co/u/Jason_Wee)\
**Post date:** [December 5, 2013, 6:57am UTC](https://discuss.elastic.co/t/high-cpu-usage-on-xdcr-transfer-to-elasticsearch-nodes/14718/2 "2013-12-05T06:57:13Z")

</div>

> **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)**
>
> Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today.

curl on your es nodes and that should give hint what is hogging the cpu.

hth

/Jason

On Thu, Dec 5, 2013 at 9:44 AM, Corey Nolet [cjnolet@gmail.com](mailto:cjnolet@gmail.com) wrote:

> I've got a 6 node HDFS cluster. Each node has 8 cores and 32gb of ram. On  
> the same nodes, I'm running Couchbase 2.2 and Elasticsearch 0.90.5 with the  
> Couchbase transport plugin 1.1.0. I have Couchbase doing real-time  
> ingest/caching from Twitter Storm and indexing the documents in  
> elasticsearch for further mining. I'm using about 18gb of ram from my  
> entire cluster for Couchbase. Without Elasticsearch running, Couchbase  
> seems to idle between 1% and 16% CPU utilization with occasional (expected)  
> spikes at around 30% to 60%. I'm okay with that.
> 
> What I'm trying to figure out, however, is why the XDCR to Elasticsearch  
> seems to pump my nodes up to a constant 60% to 99% CPU utilization. I've  
> tried lowering the max concurrent replications to 2 and even changing the  
> checkpoint sizes and I've noticed no decrease in utilization. The  
> utilization has been so high, in fact, that my Zookeeper nodes can't keep a  
> lock and they start expiring important connections.
> 
> I tried pulling Elasticsearch off of my 6 nodes and putting it elsewhere  
> (on a separate 4 nodes). This also didn't seem to have an impact. I read on  
> a slideshow that 24-cores is recommended but that's a little absurd of a  
> requirement. I'm doing about 250,000 documents every 3 minutes or so and  
> ELasticsearch seems to be keeping up. The short of it is... i really just  
> need the ability to do term queries of the documents and N1QL isn't nearly  
> as fast with the intersecting searches as I need it to be (i'm looking for  
> subsecond here).
> 
> Any ideas? I've done a ton of research and all I've seen are issues in  
> verisons \<2.2 in jira that appear to have been fixed.
> 
> Thanks!
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/a6920681-dd08-41f9-a781-c17bcefdcdb1%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a6920681-dd08-41f9-a781-c17bcefdcdb1%40googlegroups.com)  
> .  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHO4itwbtpTE7LAmiYPXV-ny2tXO15uakXpxrPojpuECUhkL1Q%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHO4itwbtpTE7LAmiYPXV-ny2tXO15uakXpxrPojpuECUhkL1Q%40mail.gmail.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [December 5, 2013, 7:14am UTC](https://discuss.elastic.co/t/high-cpu-usage-on-xdcr-transfer-to-elasticsearch-nodes/14718/3 "2013-12-05T07:14:56Z")

</div>

I did not understand if the load comes from elasticsearch process or from couchbase process?  
I guess you already check that, right?

Note: I'd isolate elasticsearch on another nodes, mainly because of IO. I guess that couchbase has a lot of reads to do and in the same time elasticsearch has a lot of writes.

--  
David 😉  
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 5 déc. 2013 à 07:57, Jason Wee [peichieh@gmail.com](mailto:peichieh@gmail.com) a écrit :

> **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)**
>
> Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today.

curl on your es nodes and that should give hint what is hogging the cpu.

hth

/Jason

> On Thu, Dec 5, 2013 at 9:44 AM, Corey Nolet [cjnolet@gmail.com](mailto:cjnolet@gmail.com) wrote:  
> I've got a 6 node HDFS cluster. Each node has 8 cores and 32gb of ram. On the same nodes, I'm running Couchbase 2.2 and Elasticsearch 0.90.5 with the Couchbase transport plugin 1.1.0. I have Couchbase doing real-time ingest/caching from Twitter Storm and indexing the documents in elasticsearch for further mining. I'm using about 18gb of ram from my entire cluster for Couchbase. Without Elasticsearch running, Couchbase seems to idle between 1% and 16% CPU utilization with occasional (expected) spikes at around 30% to 60%. I'm okay with that.
> 
> What I'm trying to figure out, however, is why the XDCR to Elasticsearch seems to pump my nodes up to a constant 60% to 99% CPU utilization. I've tried lowering the max concurrent replications to 2 and even changing the checkpoint sizes and I've noticed no decrease in utilization. The utilization has been so high, in fact, that my Zookeeper nodes can't keep a lock and they start expiring important connections.
> 
> I tried pulling Elasticsearch off of my 6 nodes and putting it elsewhere (on a separate 4 nodes). This also didn't seem to have an impact. I read on a slideshow that 24-cores is recommended but that's a little absurd of a requirement. I'm doing about 250,000 documents every 3 minutes or so and ELasticsearch seems to be keeping up. The short of it is... i really just need the ability to do term queries of the documents and N1QL isn't nearly as fast with the intersecting searches as I need it to be (i'm looking for subsecond here).
> 
> Any ideas? I've done a ton of research and all I've seen are issues in verisons \<2.2 in jira that appear to have been fixed.
> 
> Thanks!
> 
> --  
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a6920681-dd08-41f9-a781-c17bcefdcdb1%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a6920681-dd08-41f9-a781-c17bcefdcdb1%40googlegroups.com).  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHO4itwbtpTE7LAmiYPXV-ny2tXO15uakXpxrPojpuECUhkL1Q%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHO4itwbtpTE7LAmiYPXV-ny2tXO15uakXpxrPojpuECUhkL1Q%40mail.gmail.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/0552B653-6B17-4703-81C5-6ABD0A4790D1%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/0552B653-6B17-4703-81C5-6ABD0A4790D1%40pilato.fr).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Corey\_Nolet](https://avatars.discourse-cdn.com/v4/letter/c/b19c9b/32.png) [@Corey\_Nolet](https://discuss.elastic.co/u/Corey_Nolet)\
**Post date:** [December 5, 2013, 1:21pm UTC](https://discuss.elastic.co/t/high-cpu-usage-on-xdcr-transfer-to-elasticsearch-nodes/14718/4 "2013-12-05T13:21:08Z")

</div>

David,

I pulled the elasticsearch nodes from the couchbase nodes but I didn't  
notice any difference in the CPU usage on the couchbase nodes.

On Thursday, December 5, 2013 2:14:56 AM UTC-5, David Pilato wrote:

> I did not understand if the load comes from elasticsearch process or from  
> couchbase process?  
> I guess you already check that, right?
> 
> Note: I'd isolate elasticsearch on another nodes, mainly because of IO. I  
> guess that couchbase has a lot of reads to do and in the same time  
> elasticsearch has a lot of writes.
> 
> --  
> David 😉  
> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
> 
> Le 5 déc. 2013 à 07:57, Jason Wee \<[peic...@gmail.com](mailto:peic...@gmail.com) \<javascript:\>\> a  
> écrit :
> 
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-nodes-hot-threads.html)
> 
> curl on your es nodes and that should give hint what is hogging the cpu.
> 
> hth
> 
> /Jason
> 
> On Thu, Dec 5, 2013 at 9:44 AM, Corey Nolet \<[cjn...@gmail.com](mailto:cjn...@gmail.com)\<javascript:\>
> 
> > wrote:
> 
> > I've got a 6 node HDFS cluster. Each node has 8 cores and 32gb of ram. On  
> > the same nodes, I'm running Couchbase 2.2 and Elasticsearch 0.90.5 with the  
> > Couchbase transport plugin 1.1.0. I have Couchbase doing real-time  
> > ingest/caching from Twitter Storm and indexing the documents in  
> > elasticsearch for further mining. I'm using about 18gb of ram from my  
> > entire cluster for Couchbase. Without Elasticsearch running, Couchbase  
> > seems to idle between 1% and 16% CPU utilization with occasional (expected)  
> > spikes at around 30% to 60%. I'm okay with that.
> > 
> > What I'm trying to figure out, however, is why the XDCR to Elasticsearch  
> > seems to pump my nodes up to a constant 60% to 99% CPU utilization. I've  
> > tried lowering the max concurrent replications to 2 and even changing the  
> > checkpoint sizes and I've noticed no decrease in utilization. The  
> > utilization has been so high, in fact, that my Zookeeper nodes can't keep a  
> > lock and they start expiring important connections.
> > 
> > I tried pulling Elasticsearch off of my 6 nodes and putting it elsewhere  
> > (on a separate 4 nodes). This also didn't seem to have an impact. I read on  
> > a slideshow that 24-cores is recommended but that's a little absurd of a  
> > requirement. I'm doing about 250,000 documents every 3 minutes or so and  
> > ELasticsearch seems to be keeping up. The short of it is... i really just  
> > need the ability to do term queries of the documents and N1QL isn't nearly  
> > as fast with the intersecting searches as I need it to be (i'm looking for  
> > subsecond here).
> > 
> > Any ideas? I've done a ton of research and all I've seen are issues in  
> > verisons \<2.2 in jira that appear to have been fixed.
> > 
> > Thanks!
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> > To view this discussion on the web visit  
> > [https://groups.google.com/d/msgid/elasticsearch/a6920681-dd08-41f9-a781-c17bcefdcdb1%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a6920681-dd08-41f9-a781-c17bcefdcdb1%40googlegroups.com)  
> > .  
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/CAHO4itwbtpTE7LAmiYPXV-ny2tXO15uakXpxrPojpuECUhkL1Q%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHO4itwbtpTE7LAmiYPXV-ny2tXO15uakXpxrPojpuECUhkL1Q%40mail.gmail.com)  
> .  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/7aae5306-cbd1-45c8-83f6-9f02533101b9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7aae5306-cbd1-45c8-83f6-9f02533101b9%40googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [December 5, 2013, 1:43pm UTC](https://discuss.elastic.co/t/high-cpu-usage-on-xdcr-transfer-to-elasticsearch-nodes/14718/5 "2013-12-05T13:43:37Z")

</div>

So you mean that CPU usage is high on Couchbase nodes but not on elasticsearch, right?  
Or is the high CPU usage on elasticsearch nodes?

Did you try to run the hot threads command suggested by Jason?

What are your elasticsearch settings? How much RAM did you give to the heap?

--  
David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com)  
@dadoonet | @elasticsearchfr

Le 5 décembre 2013 at 14:21:11, Corey Nolet ([cjnolet@gmail.com](mailto:cjnolet@gmail.com)) a écrit:

David,

I pulled the elasticsearch nodes from the couchbase nodes but I didn't notice any difference in the CPU usage on the couchbase nodes.

On Thursday, December 5, 2013 2:14:56 AM UTC-5, David Pilato wrote:  
I did not understand if the load comes from elasticsearch process or from couchbase process?  
I guess you already check that, right?

Note: I'd isolate elasticsearch on another nodes, mainly because of IO. I guess that couchbase has a lot of reads to do and in the same time elasticsearch has a lot of writes.

--  
David 😉  
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 5 déc. 2013 à 07:57, Jason Wee [peic...@gmail.com](mailto:peic...@gmail.com) a écrit :

[http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-nodes-hot-threads.html](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-nodes-hot-threads.html)

curl on your es nodes and that should give hint what is hogging the cpu.

hth

/Jason

On Thu, Dec 5, 2013 at 9:44 AM, Corey Nolet [cjn...@gmail.com](mailto:cjn...@gmail.com) wrote:  
I've got a 6 node HDFS cluster. Each node has 8 cores and 32gb of ram. On the same nodes, I'm running Couchbase 2.2 and Elasticsearch 0.90.5 with the Couchbase transport plugin 1.1.0. I have Couchbase doing real-time ingest/caching from Twitter Storm and indexing the documents in elasticsearch for further mining. I'm using about 18gb of ram from my entire cluster for Couchbase. Without ElasticSearch running, Couchbase seems to idle between 1% and 16% CPU utilization with occasional (expected) spikes at around 30% to 60%. I'm okay with that.

What I'm trying to figure out, however, is why the XDCR to ElasticSearch seems to pump my nodes up to a constant 60% to 99% CPU utilization. I've tried lowering the max concurrent replications to 2 and even changing the checkpoint sizes and I've noticed no decrease in utilization. The utilization has been so high, in fact, that my Zookeeper nodes can't keep a lock and they start expiring important connections.

I tried pulling Elasticsearch off of my 6 nodes and putting it elsewhere (on a separate 4 nodes). This also didn't seem to have an impact. I read on a slideshow that 24-cores is recommended but that's a little absurd of a requirement. I'm doing about 250,000 documents every 3 minutes or so and ELasticsearch seems to be keeping up. The short of it is... i really just need the ability to do term queries of the documents and N1QL isn't nearly as fast with the intersecting searches as I need it to be (i'm looking for subsecond here).

Any ideas? I've done a ton of research and all I've seen are issues in verisons \<2.2 in jira that appear to have been fixed.

Thanks!

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a6920681-dd08-41f9-a781-c17bcefdcdb1%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a6920681-dd08-41f9-a781-c17bcefdcdb1%40googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

## -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHO4itwbtpTE7LAmiYPXV-ny2tXO15uakXpxrPojpuECUhkL1Q%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHO4itwbtpTE7LAmiYPXV-ny2tXO15uakXpxrPojpuECUhkL1Q%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/7aae5306-cbd1-45c8-83f6-9f02533101b9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7aae5306-cbd1-45c8-83f6-9f02533101b9%40googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/etPan.52a08309.14330624.bd3d%40MacBook-Air-de-David.local](https://groups.google.com/d/msgid/elasticsearch/etPan.52a08309.14330624.bd3d%40MacBook-Air-de-David.local).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Corey\_Nolet](https://avatars.discourse-cdn.com/v4/letter/c/b19c9b/32.png) [@Corey\_Nolet](https://discuss.elastic.co/u/Corey_Nolet)\
**Post date:** [December 6, 2013, 4:25am UTC](https://discuss.elastic.co/t/high-cpu-usage-on-xdcr-transfer-to-elasticsearch-nodes/14718/6 "2013-12-06T04:25:16Z")

</div>

From my tinkering and troubleshooting, what I've noticed is that the  
couchbase cluster's CPU utilization is high when it has XDCR enabled to  
push to the elasticsearch cluster. The elasticsearch cluster had moderate  
CPU usage but nothing like the couchbase cluster.

I'll tinker a bit more at work tomorrow and try to get more definitive  
results. I'd also like to try the hot threads call to see if I can isolate  
it further.

On Thursday, December 5, 2013 8:43:37 AM UTC-5, David Pilato wrote:

> So you mean that CPU usage is high on Couchbase nodes but not on  
> elasticsearch, right?  
> Or is the high CPU usage on elasticsearch nodes?
> 
> Did you try to run the hot threads command suggested by Jason?
> 
> What are your elasticsearch settings? How much RAM did you give to the  
> heap?
> 
> --  
> _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)_  
> @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr[https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr)
> 
> Le 5 décembre 2013 at 14:21:11, Corey Nolet ([cjn...@gmail.com](mailto:cjn...@gmail.com)\<javascript:\>)  
> a écrit:
> 
> David,
> 
> I pulled the elasticsearch nodes from the couchbase nodes but I didn't  
> notice any difference in the CPU usage on the couchbase nodes.
> 
> On Thursday, December 5, 2013 2:14:56 AM UTC-5, David Pilato wrote:
> 
> > I did not understand if the load comes from elasticsearch process or  
> > from couchbase process?  
> > I guess you already check that, right?
> > 
> > Note: I'd isolate elasticsearch on another nodes, mainly because of IO. I  
> > guess that couchbase has a lot of reads to do and in the same time  
> > elasticsearch has a lot of writes.
> > 
> > --  
> > David 😉  
> > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
> > 
> > Le 5 déc. 2013 à 07:57, Jason Wee [peic...@gmail.com](mailto:peic...@gmail.com) a écrit :
> > 
> > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-nodes-hot-threads.html)
> > 
> > curl on your es nodes and that should give hint what is hogging the cpu.
> > 
> > hth
> > 
> > /Jason
> > 
> > On Thu, Dec 5, 2013 at 9:44 AM, Corey Nolet [cjn...@gmail.com](mailto:cjn...@gmail.com) wrote:
> > 
> > > I've got a 6 node HDFS cluster. Each node has 8 cores and 32gb of ram.  
> > > On the same nodes, I'm running Couchbase 2.2 and Elasticsearch 0.90.5 with  
> > > the Couchbase transport plugin 1.1.0. I have Couchbase doing real-time  
> > > ingest/caching from Twitter Storm and indexing the documents in  
> > > elasticsearch for further mining. I'm using about 18gb of ram from my  
> > > entire cluster for Couchbase. Without Elasticsearch running, Couchbase  
> > > seems to idle between 1% and 16% CPU utilization with occasional (expected)  
> > > spikes at around 30% to 60%. I'm okay with that.
> > > 
> > > What I'm trying to figure out, however, is why the XDCR to Elasticsearch  
> > > seems to pump my nodes up to a constant 60% to 99% CPU utilization. I've  
> > > tried lowering the max concurrent replications to 2 and even changing the  
> > > checkpoint sizes and I've noticed no decrease in utilization. The  
> > > utilization has been so high, in fact, that my Zookeeper nodes can't keep a  
> > > lock and they start expiring important connections.
> > > 
> > > I tried pulling Elasticsearch off of my 6 nodes and putting it elsewhere  
> > > (on a separate 4 nodes). This also didn't seem to have an impact. I read on  
> > > a slideshow that 24-cores is recommended but that's a little absurd of a  
> > > requirement. I'm doing about 250,000 documents every 3 minutes or so and  
> > > ELasticsearch seems to be keeping up. The short of it is... i really just  
> > > need the ability to do term queries of the documents and N1QL isn't nearly  
> > > as fast with the intersecting searches as I need it to be (i'm looking for  
> > > subsecond here).
> > > 
> > > Any ideas? I've done a ton of research and all I've seen are issues in  
> > > verisons \<2.2 in jira that appear to have been fixed.
> > > 
> > > Thanks!
> > > 
> > > --  
> > > You received this message because you are subscribed to the Google  
> > > Groups "elasticsearch" group.  
> > > To unsubscribe from this group and stop receiving emails from it, send  
> > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > To view this discussion on the web visit  
> > > [https://groups.google.com/d/msgid/elasticsearch/a6920681-dd08-41f9-a781-c17bcefdcdb1%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a6920681-dd08-41f9-a781-c17bcefdcdb1%40googlegroups.com)  
> > > .  
> > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > To view this discussion on the web visit  
> > [https://groups.google.com/d/msgid/elasticsearch/CAHO4itwbtpTE7LAmiYPXV-ny2tXO15uakXpxrPojpuECUhkL1Q%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHO4itwbtpTE7LAmiYPXV-ny2tXO15uakXpxrPojpuECUhkL1Q%40mail.gmail.com)  
> > .  
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/7aae5306-cbd1-45c8-83f6-9f02533101b9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7aae5306-cbd1-45c8-83f6-9f02533101b9%40googlegroups.com)  
> .  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/da08893e-8212-4276-8c2b-5e933cf7276a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/da08893e-8212-4276-8c2b-5e933cf7276a%40googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [December 6, 2013, 7:07am UTC](https://discuss.elastic.co/t/high-cpu-usage-on-xdcr-transfer-to-elasticsearch-nodes/14718/7 "2013-12-06T07:07:07Z")

</div>

If Elasticsearch CPU usage is not so high, then hot\_threads won't help here.  
I think you should open an issue in couchbase plugin project or ask to the couchbase mailing list.

My 2 cents.

--  
David 😉  
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 6 déc. 2013 à 05:25, Corey Nolet [cjnolet@gmail.com](mailto:cjnolet@gmail.com) a écrit :

From my tinkering and troubleshooting, what I've noticed is that the couchbase cluster's CPU utilization is high when it has XDCR enabled to push to the elasticsearch cluster. The elasticsearch cluster had moderate CPU usage but nothing like the couchbase cluster.

I'll tinker a bit more at work tomorrow and try to get more definitive results. I'd also like to try the hot threads call to see if I can isolate it further.

> On Thursday, December 5, 2013 8:43:37 AM UTC-5, David Pilato wrote:  
> So you mean that CPU usage is high on Couchbase nodes but not on elasticsearch, right?  
> Or is the high CPU usage on elasticsearch nodes?
> 
> Did you try to run the hot threads command suggested by Jason?
> 
> What are your elasticsearch settings? How much RAM did you give to the heap?
> 
> --  
> David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com)  
> @dadoonet | @elasticsearchfr
> 
> Le 5 décembre 2013 at 14:21:11, Corey Nolet ([cjn...@gmail.com](mailto:cjn...@gmail.com)) a écrit:
> 
> > David,
> > 
> > I pulled the elasticsearch nodes from the couchbase nodes but I didn't notice any difference in the CPU usage on the couchbase nodes.
> > 
> > > On Thursday, December 5, 2013 2:14:56 AM UTC-5, David Pilato wrote:  
> > > I did not understand if the load comes from elasticsearch process or from couchbase process?  
> > > I guess you already check that, right?
> > > 
> > > Note: I'd isolate elasticsearch on another nodes, mainly because of IO. I guess that couchbase has a lot of reads to do and in the same time elasticsearch has a lot of writes.
> > > 
> > > --  
> > > David 😉  
> > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
> > > 
> > > Le 5 déc. 2013 à 07:57, Jason Wee [peic...@gmail.com](mailto:peic...@gmail.com) a écrit :
> > > 
> > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-nodes-hot-threads.html)
> > > 
> > > curl on your es nodes and that should give hint what is hogging the cpu.
> > > 
> > > hth
> > > 
> > > /Jason
> > > 
> > > > On Thu, Dec 5, 2013 at 9:44 AM, Corey Nolet [cjn...@gmail.com](mailto:cjn...@gmail.com) wrote:  
> > > > I've got a 6 node HDFS cluster. Each node has 8 cores and 32gb of ram. On the same nodes, I'm running Couchbase 2.2 and Elasticsearch 0.90.5 with the Couchbase transport plugin 1.1.0. I have Couchbase doing real-time ingest/caching from Twitter Storm and indexing the documents in elasticsearch for further mining. I'm using about 18gb of ram from my entire cluster for Couchbase. Without Elasticsearch running, Couchbase seems to idle between 1% and 16% CPU utilization with occasional (expected) spikes at around 30% to 60%. I'm okay with that.
> > > > 
> > > > What I'm trying to figure out, however, is why the XDCR to Elasticsearch seems to pump my nodes up to a constant 60% to 99% CPU utilization. I've tried lowering the max concurrent replications to 2 and even changing the checkpoint sizes and I've noticed no decrease in utilization. The utilization has been so high, in fact, that my Zookeeper nodes can't keep a lock and they start expiring important connections.
> > > > 
> > > > I tried pulling Elasticsearch off of my 6 nodes and putting it elsewhere (on a separate 4 nodes). This also didn't seem to have an impact. I read on a slideshow that 24-cores is recommended but that's a little absurd of a requirement. I'm doing about 250,000 documents every 3 minutes or so and ELasticsearch seems to be keeping up. The short of it is... i really just need the ability to do term queries of the documents and N1QL isn't nearly as fast with the intersecting searches as I need it to be (i'm looking for subsecond here).
> > > > 
> > > > Any ideas? I've done a ton of research and all I've seen are issues in verisons \<2.2 in jira that appear to have been fixed.
> > > > 
> > > > Thanks!
> > > > 
> > > > --  
> > > > You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> > > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a6920681-dd08-41f9-a781-c17bcefdcdb1%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a6920681-dd08-41f9-a781-c17bcefdcdb1%40googlegroups.com).  
> > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> > > 
> > > --  
> > > You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHO4itwbtpTE7LAmiYPXV-ny2tXO15uakXpxrPojpuECUhkL1Q%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHO4itwbtpTE7LAmiYPXV-ny2tXO15uakXpxrPojpuECUhkL1Q%40mail.gmail.com).  
> > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).  
> > > --  
> > > You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/7aae5306-cbd1-45c8-83f6-9f02533101b9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7aae5306-cbd1-45c8-83f6-9f02533101b9%40googlegroups.com).  
> > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/da08893e-8212-4276-8c2b-5e933cf7276a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/da08893e-8212-4276-8c2b-5e933cf7276a%40googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/AB3E8DDB-FCFD-4D2D-87B3-1E9ABD4EF237%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/AB3E8DDB-FCFD-4D2D-87B3-1E9ABD4EF237%40pilato.fr).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 2:02am UTC](https://discuss.elastic.co/t/high-cpu-usage-on-xdcr-transfer-to-elasticsearch-nodes/14718/8 "2017-07-06T02:02:55Z")

</div>