High CPU Usage on XDCR Transfer to ElasticSearch nodes


(Corey Nolet) #1

I've got a 6 node HDFS cluster. Each node has 8 cores and 32gb of ram. On
the same nodes, I'm running Couchbase 2.2 and Elasticsearch 0.90.5 with the
Couchbase transport plugin 1.1.0. I have Couchbase doing real-time
ingest/caching from Twitter Storm and indexing the documents in
elasticsearch for further mining. I'm using about 18gb of ram from my
entire cluster for Couchbase. Without ElasticSearch running, Couchbase
seems to idle between 1% and 16% CPU utilization with occasional (expected)
spikes at around 30% to 60%. I'm okay with that.

What I'm trying to figure out, however, is why the XDCR to ElasticSearch
seems to pump my nodes up to a constant 60% to 99% CPU utilization. I've
tried lowering the max concurrent replications to 2 and even changing the
checkpoint sizes and I've noticed no decrease in utilization. The
utilization has been so high, in fact, that my Zookeeper nodes can't keep a
lock and they start expiring important connections.

I tried pulling Elasticsearch off of my 6 nodes and putting it elsewhere
(on a separate 4 nodes). This also didn't seem to have an impact. I read on
a slideshow that 24-cores is recommended but that's a little absurd of a
requirement. I'm doing about 250,000 documents every 3 minutes or so and
ELasticsearch seems to be keeping up. The short of it is... i really just
need the ability to do term queries of the documents and N1QL isn't nearly
as fast with the intersecting searches as I need it to be (i'm looking for
subsecond here).

Any ideas? I've done a ton of research and all I've seen are issues in
verisons <2.2 in jira that appear to have been fixed.

Thanks!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a6920681-dd08-41f9-a781-c17bcefdcdb1%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Jason Wee) #2

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-nodes-hot-threads.html

curl on your es nodes and that should give hint what is hogging the cpu.

hth

/Jason

On Thu, Dec 5, 2013 at 9:44 AM, Corey Nolet cjnolet@gmail.com wrote:

I've got a 6 node HDFS cluster. Each node has 8 cores and 32gb of ram. On
the same nodes, I'm running Couchbase 2.2 and Elasticsearch 0.90.5 with the
Couchbase transport plugin 1.1.0. I have Couchbase doing real-time
ingest/caching from Twitter Storm and indexing the documents in
elasticsearch for further mining. I'm using about 18gb of ram from my
entire cluster for Couchbase. Without ElasticSearch running, Couchbase
seems to idle between 1% and 16% CPU utilization with occasional (expected)
spikes at around 30% to 60%. I'm okay with that.

What I'm trying to figure out, however, is why the XDCR to ElasticSearch
seems to pump my nodes up to a constant 60% to 99% CPU utilization. I've
tried lowering the max concurrent replications to 2 and even changing the
checkpoint sizes and I've noticed no decrease in utilization. The
utilization has been so high, in fact, that my Zookeeper nodes can't keep a
lock and they start expiring important connections.

I tried pulling Elasticsearch off of my 6 nodes and putting it elsewhere
(on a separate 4 nodes). This also didn't seem to have an impact. I read on
a slideshow that 24-cores is recommended but that's a little absurd of a
requirement. I'm doing about 250,000 documents every 3 minutes or so and
ELasticsearch seems to be keeping up. The short of it is... i really just
need the ability to do term queries of the documents and N1QL isn't nearly
as fast with the intersecting searches as I need it to be (i'm looking for
subsecond here).

Any ideas? I've done a ton of research and all I've seen are issues in
verisons <2.2 in jira that appear to have been fixed.

Thanks!

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/a6920681-dd08-41f9-a781-c17bcefdcdb1%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHO4itwbtpTE7LAmiYPXV-ny2tXO15uakXpxrPojpuECUhkL1Q%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(David Pilato) #3

I did not understand if the load comes from elasticsearch process or from couchbase process?
I guess you already check that, right?

Note: I'd isolate elasticsearch on another nodes, mainly because of IO. I guess that couchbase has a lot of reads to do and in the same time elasticsearch has a lot of writes.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 5 déc. 2013 à 07:57, Jason Wee peichieh@gmail.com a écrit :

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-nodes-hot-threads.html

curl on your es nodes and that should give hint what is hogging the cpu.

hth

/Jason

On Thu, Dec 5, 2013 at 9:44 AM, Corey Nolet cjnolet@gmail.com wrote:
I've got a 6 node HDFS cluster. Each node has 8 cores and 32gb of ram. On the same nodes, I'm running Couchbase 2.2 and Elasticsearch 0.90.5 with the Couchbase transport plugin 1.1.0. I have Couchbase doing real-time ingest/caching from Twitter Storm and indexing the documents in elasticsearch for further mining. I'm using about 18gb of ram from my entire cluster for Couchbase. Without ElasticSearch running, Couchbase seems to idle between 1% and 16% CPU utilization with occasional (expected) spikes at around 30% to 60%. I'm okay with that.

What I'm trying to figure out, however, is why the XDCR to ElasticSearch seems to pump my nodes up to a constant 60% to 99% CPU utilization. I've tried lowering the max concurrent replications to 2 and even changing the checkpoint sizes and I've noticed no decrease in utilization. The utilization has been so high, in fact, that my Zookeeper nodes can't keep a lock and they start expiring important connections.

I tried pulling Elasticsearch off of my 6 nodes and putting it elsewhere (on a separate 4 nodes). This also didn't seem to have an impact. I read on a slideshow that 24-cores is recommended but that's a little absurd of a requirement. I'm doing about 250,000 documents every 3 minutes or so and ELasticsearch seems to be keeping up. The short of it is... i really just need the ability to do term queries of the documents and N1QL isn't nearly as fast with the intersecting searches as I need it to be (i'm looking for subsecond here).

Any ideas? I've done a ton of research and all I've seen are issues in verisons <2.2 in jira that appear to have been fixed.

Thanks!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a6920681-dd08-41f9-a781-c17bcefdcdb1%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHO4itwbtpTE7LAmiYPXV-ny2tXO15uakXpxrPojpuECUhkL1Q%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0552B653-6B17-4703-81C5-6ABD0A4790D1%40pilato.fr.
For more options, visit https://groups.google.com/groups/opt_out.


(Corey Nolet) #4

David,

I pulled the elasticsearch nodes from the couchbase nodes but I didn't
notice any difference in the CPU usage on the couchbase nodes.

On Thursday, December 5, 2013 2:14:56 AM UTC-5, David Pilato wrote:

I did not understand if the load comes from elasticsearch process or from
couchbase process?
I guess you already check that, right?

Note: I'd isolate elasticsearch on another nodes, mainly because of IO. I
guess that couchbase has a lot of reads to do and in the same time
elasticsearch has a lot of writes.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 5 déc. 2013 à 07:57, Jason Wee <peic...@gmail.com <javascript:>> a
écrit :

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-nodes-hot-threads.html

curl on your es nodes and that should give hint what is hogging the cpu.

hth

/Jason

On Thu, Dec 5, 2013 at 9:44 AM, Corey Nolet <cjn...@gmail.com<javascript:>

wrote:

I've got a 6 node HDFS cluster. Each node has 8 cores and 32gb of ram. On
the same nodes, I'm running Couchbase 2.2 and Elasticsearch 0.90.5 with the
Couchbase transport plugin 1.1.0. I have Couchbase doing real-time
ingest/caching from Twitter Storm and indexing the documents in
elasticsearch for further mining. I'm using about 18gb of ram from my
entire cluster for Couchbase. Without ElasticSearch running, Couchbase
seems to idle between 1% and 16% CPU utilization with occasional (expected)
spikes at around 30% to 60%. I'm okay with that.

What I'm trying to figure out, however, is why the XDCR to ElasticSearch
seems to pump my nodes up to a constant 60% to 99% CPU utilization. I've
tried lowering the max concurrent replications to 2 and even changing the
checkpoint sizes and I've noticed no decrease in utilization. The
utilization has been so high, in fact, that my Zookeeper nodes can't keep a
lock and they start expiring important connections.

I tried pulling Elasticsearch off of my 6 nodes and putting it elsewhere
(on a separate 4 nodes). This also didn't seem to have an impact. I read on
a slideshow that 24-cores is recommended but that's a little absurd of a
requirement. I'm doing about 250,000 documents every 3 minutes or so and
ELasticsearch seems to be keeping up. The short of it is... i really just
need the ability to do term queries of the documents and N1QL isn't nearly
as fast with the intersecting searches as I need it to be (i'm looking for
subsecond here).

Any ideas? I've done a ton of research and all I've seen are issues in
verisons <2.2 in jira that appear to have been fixed.

Thanks!

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/a6920681-dd08-41f9-a781-c17bcefdcdb1%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHO4itwbtpTE7LAmiYPXV-ny2tXO15uakXpxrPojpuECUhkL1Q%40mail.gmail.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7aae5306-cbd1-45c8-83f6-9f02533101b9%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(David Pilato) #5

So you mean that CPU usage is high on Couchbase nodes but not on elasticsearch, right?
Or is the high CPU usage on elasticsearch nodes?

Did you try to run the hot threads command suggested by Jason?

What are your elasticsearch settings? How much RAM did you give to the heap?

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 5 décembre 2013 at 14:21:11, Corey Nolet (cjnolet@gmail.com) a écrit:

David,

I pulled the elasticsearch nodes from the couchbase nodes but I didn't notice any difference in the CPU usage on the couchbase nodes.

On Thursday, December 5, 2013 2:14:56 AM UTC-5, David Pilato wrote:
I did not understand if the load comes from elasticsearch process or from couchbase process?
I guess you already check that, right?

Note: I'd isolate elasticsearch on another nodes, mainly because of IO. I guess that couchbase has a lot of reads to do and in the same time elasticsearch has a lot of writes.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 5 déc. 2013 à 07:57, Jason Wee peic...@gmail.com a écrit :

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-nodes-hot-threads.html

curl on your es nodes and that should give hint what is hogging the cpu.

hth

/Jason

On Thu, Dec 5, 2013 at 9:44 AM, Corey Nolet cjn...@gmail.com wrote:
I've got a 6 node HDFS cluster. Each node has 8 cores and 32gb of ram. On the same nodes, I'm running Couchbase 2.2 and Elasticsearch 0.90.5 with the Couchbase transport plugin 1.1.0. I have Couchbase doing real-time ingest/caching from Twitter Storm and indexing the documents in elasticsearch for further mining. I'm using about 18gb of ram from my entire cluster for Couchbase. Without ElasticSearch running, Couchbase seems to idle between 1% and 16% CPU utilization with occasional (expected) spikes at around 30% to 60%. I'm okay with that.

What I'm trying to figure out, however, is why the XDCR to ElasticSearch seems to pump my nodes up to a constant 60% to 99% CPU utilization. I've tried lowering the max concurrent replications to 2 and even changing the checkpoint sizes and I've noticed no decrease in utilization. The utilization has been so high, in fact, that my Zookeeper nodes can't keep a lock and they start expiring important connections.

I tried pulling Elasticsearch off of my 6 nodes and putting it elsewhere (on a separate 4 nodes). This also didn't seem to have an impact. I read on a slideshow that 24-cores is recommended but that's a little absurd of a requirement. I'm doing about 250,000 documents every 3 minutes or so and ELasticsearch seems to be keeping up. The short of it is... i really just need the ability to do term queries of the documents and N1QL isn't nearly as fast with the intersecting searches as I need it to be (i'm looking for subsecond here).

Any ideas? I've done a ton of research and all I've seen are issues in verisons <2.2 in jira that appear to have been fixed.

Thanks!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a6920681-dd08-41f9-a781-c17bcefdcdb1%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHO4itwbtpTE7LAmiYPXV-ny2tXO15uakXpxrPojpuECUhkL1Q%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7aae5306-cbd1-45c8-83f6-9f02533101b9%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.52a08309.14330624.bd3d%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/groups/opt_out.


(Corey Nolet) #6

From my tinkering and troubleshooting, what I've noticed is that the
couchbase cluster's CPU utilization is high when it has XDCR enabled to
push to the elasticsearch cluster. The elasticsearch cluster had moderate
CPU usage but nothing like the couchbase cluster.

I'll tinker a bit more at work tomorrow and try to get more definitive
results. I'd also like to try the hot threads call to see if I can isolate
it further.

On Thursday, December 5, 2013 8:43:37 AM UTC-5, David Pilato wrote:

So you mean that CPU usage is high on Couchbase nodes but not on
elasticsearch, right?
Or is the high CPU usage on elasticsearch nodes?

Did you try to run the hot threads command suggested by Jason?

What are your elasticsearch settings? How much RAM did you give to the
heap?

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr

Le 5 décembre 2013 at 14:21:11, Corey Nolet (cjn...@gmail.com<javascript:>)
a écrit:

David,

I pulled the elasticsearch nodes from the couchbase nodes but I didn't
notice any difference in the CPU usage on the couchbase nodes.

On Thursday, December 5, 2013 2:14:56 AM UTC-5, David Pilato wrote:

I did not understand if the load comes from elasticsearch process or
from couchbase process?
I guess you already check that, right?

Note: I'd isolate elasticsearch on another nodes, mainly because of IO. I
guess that couchbase has a lot of reads to do and in the same time
elasticsearch has a lot of writes.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 5 déc. 2013 à 07:57, Jason Wee peic...@gmail.com a écrit :

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-nodes-hot-threads.html

curl on your es nodes and that should give hint what is hogging the cpu.

hth

/Jason

On Thu, Dec 5, 2013 at 9:44 AM, Corey Nolet cjn...@gmail.com wrote:

I've got a 6 node HDFS cluster. Each node has 8 cores and 32gb of ram.
On the same nodes, I'm running Couchbase 2.2 and Elasticsearch 0.90.5 with
the Couchbase transport plugin 1.1.0. I have Couchbase doing real-time
ingest/caching from Twitter Storm and indexing the documents in
elasticsearch for further mining. I'm using about 18gb of ram from my
entire cluster for Couchbase. Without ElasticSearch running, Couchbase
seems to idle between 1% and 16% CPU utilization with occasional (expected)
spikes at around 30% to 60%. I'm okay with that.

What I'm trying to figure out, however, is why the XDCR to ElasticSearch
seems to pump my nodes up to a constant 60% to 99% CPU utilization. I've
tried lowering the max concurrent replications to 2 and even changing the
checkpoint sizes and I've noticed no decrease in utilization. The
utilization has been so high, in fact, that my Zookeeper nodes can't keep a
lock and they start expiring important connections.

I tried pulling Elasticsearch off of my 6 nodes and putting it elsewhere
(on a separate 4 nodes). This also didn't seem to have an impact. I read on
a slideshow that 24-cores is recommended but that's a little absurd of a
requirement. I'm doing about 250,000 documents every 3 minutes or so and
ELasticsearch seems to be keeping up. The short of it is... i really just
need the ability to do term queries of the documents and N1QL isn't nearly
as fast with the intersecting searches as I need it to be (i'm looking for
subsecond here).

Any ideas? I've done a ton of research and all I've seen are issues in
verisons <2.2 in jira that appear to have been fixed.

Thanks!

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/a6920681-dd08-41f9-a781-c17bcefdcdb1%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHO4itwbtpTE7LAmiYPXV-ny2tXO15uakXpxrPojpuECUhkL1Q%40mail.gmail.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/7aae5306-cbd1-45c8-83f6-9f02533101b9%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/da08893e-8212-4276-8c2b-5e933cf7276a%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(David Pilato) #7

If Elasticsearch CPU usage is not so high, then hot_threads won't help here.
I think you should open an issue in couchbase plugin project or ask to the couchbase mailing list.

My 2 cents.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 6 déc. 2013 à 05:25, Corey Nolet cjnolet@gmail.com a écrit :

From my tinkering and troubleshooting, what I've noticed is that the couchbase cluster's CPU utilization is high when it has XDCR enabled to push to the elasticsearch cluster. The elasticsearch cluster had moderate CPU usage but nothing like the couchbase cluster.

I'll tinker a bit more at work tomorrow and try to get more definitive results. I'd also like to try the hot threads call to see if I can isolate it further.

On Thursday, December 5, 2013 8:43:37 AM UTC-5, David Pilato wrote:
So you mean that CPU usage is high on Couchbase nodes but not on elasticsearch, right?
Or is the high CPU usage on elasticsearch nodes?

Did you try to run the hot threads command suggested by Jason?

What are your elasticsearch settings? How much RAM did you give to the heap?

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 5 décembre 2013 at 14:21:11, Corey Nolet (cjn...@gmail.com) a écrit:

David,

I pulled the elasticsearch nodes from the couchbase nodes but I didn't notice any difference in the CPU usage on the couchbase nodes.

On Thursday, December 5, 2013 2:14:56 AM UTC-5, David Pilato wrote:
I did not understand if the load comes from elasticsearch process or from couchbase process?
I guess you already check that, right?

Note: I'd isolate elasticsearch on another nodes, mainly because of IO. I guess that couchbase has a lot of reads to do and in the same time elasticsearch has a lot of writes.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 5 déc. 2013 à 07:57, Jason Wee peic...@gmail.com a écrit :

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-nodes-hot-threads.html

curl on your es nodes and that should give hint what is hogging the cpu.

hth

/Jason

On Thu, Dec 5, 2013 at 9:44 AM, Corey Nolet cjn...@gmail.com wrote:
I've got a 6 node HDFS cluster. Each node has 8 cores and 32gb of ram. On the same nodes, I'm running Couchbase 2.2 and Elasticsearch 0.90.5 with the Couchbase transport plugin 1.1.0. I have Couchbase doing real-time ingest/caching from Twitter Storm and indexing the documents in elasticsearch for further mining. I'm using about 18gb of ram from my entire cluster for Couchbase. Without ElasticSearch running, Couchbase seems to idle between 1% and 16% CPU utilization with occasional (expected) spikes at around 30% to 60%. I'm okay with that.

What I'm trying to figure out, however, is why the XDCR to ElasticSearch seems to pump my nodes up to a constant 60% to 99% CPU utilization. I've tried lowering the max concurrent replications to 2 and even changing the checkpoint sizes and I've noticed no decrease in utilization. The utilization has been so high, in fact, that my Zookeeper nodes can't keep a lock and they start expiring important connections.

I tried pulling Elasticsearch off of my 6 nodes and putting it elsewhere (on a separate 4 nodes). This also didn't seem to have an impact. I read on a slideshow that 24-cores is recommended but that's a little absurd of a requirement. I'm doing about 250,000 documents every 3 minutes or so and ELasticsearch seems to be keeping up. The short of it is... i really just need the ability to do term queries of the documents and N1QL isn't nearly as fast with the intersecting searches as I need it to be (i'm looking for subsecond here).

Any ideas? I've done a ton of research and all I've seen are issues in verisons <2.2 in jira that appear to have been fixed.

Thanks!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a6920681-dd08-41f9-a781-c17bcefdcdb1%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHO4itwbtpTE7LAmiYPXV-ny2tXO15uakXpxrPojpuECUhkL1Q%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7aae5306-cbd1-45c8-83f6-9f02533101b9%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/da08893e-8212-4276-8c2b-5e933cf7276a%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/AB3E8DDB-FCFD-4D2D-87B3-1E9ABD4EF237%40pilato.fr.
For more options, visit https://groups.google.com/groups/opt_out.


(system) #8