# Distribution of work
**URL:** https://discuss.elastic.co/t/distribution-of-work/7231
**Category:** Elasticsearch
**Created:** [April 4, 2012, 7:49pm UTC](https://discuss.elastic.co/t/distribution-of-work/7231 "2012-04-04T19:49:05Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@orenmazor](https://discuss.elastic.co/u/orenmazor)
#### Post date: [April 4, 2012, 7:49pm UTC](https://discuss.elastic.co/t/distribution-of-work/7231/1 "2012-04-04T19:49:05Z")
hey guys,
I've been on/off dealing with some ES issues, and I think I've
resolved a LOT of them. but there's still one thing that bugs me.
My structure:
40gb index
two nodes, ten shards, one replica.
each node is a quad core 32gb of ram machine. min=max=16gb
our data is extremely uniform, and extremely small. but comes in at a
high rate of around 100 index actions per second.
ES is at 18.6 (I know, we'll be updating this week)
currently, all of the shards are marked as primary on node one.
node 1:
uptime is 9 hours.
big desk shows 11399 garbage collection actions, and 2000 merges.
cpu has been uniform at under 5% for pretty much ever.
memory is at used=30gb, actual=14gb
node 2:
uptime is 3 hours
bigdesk shows 104000 gc actions, and 800 merges.
cpu has been uniform at 80-90% for pretty much forever.
memory is at used=21gb, actual=13gb
note:
node 1 is also running a few python scripts (server density, mongo
sync tool, geckoboard agent)
my questions:
- why is cpu so high on node 2?
- I was thinking that the 'primary' shards are the ones that do the
merging, and the 'slaves' do the receives of the complete index, but
in both cases lucene indices have to do a merge. not so much a
question as making sure I'm understanding everything.
- my data is routed, which in theory should mean that some shards
would be doing a lot more work than others. does ES account for 'work'
and move shards from machine to machine? or does it just account for
shard size?
thanks!
Oren
---
### Author:  [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)
#### Post date: [April 4, 2012, 9:12pm UTC](https://discuss.elastic.co/t/distribution-of-work/7231/2 "2012-04-04T21:12:04Z")
Hi,
I probably do not have answers to all your questions but let me try to
address some of them.
1. As for the high CPU, can this be caused by the fact that you connect ES
client just to he second node? Which means that this node needs to handle
query execution as well? In other words, you did not share with us how you
query the cluster. It could be one of the reasons.
2. ES replicates documents, not Lucene indices segments. For more about
rationale behind this approach you can check Shay's talk from BBUZZ 2011:
[http://vimeo.com/26710663](http://vimeo.com/26710663)
Slides can be found here:
[http://2011.berlinbuzzwords.de/sites/2011.berlinbuzzwords.de/files/elasticsearch-bbuzz2011.pdf](http://2011.berlinbuzzwords.de/sites/2011.berlinbuzzwords.de/files/elasticsearch-bbuzz2011.pdf)
There is also come older draft of concept pages based on this talk sitting
in pull request. Specifically, the replication part is here:
[https://github.com/lukas-vlcek/elasticsearch.github.com/blob/cd92cb1cfaa238d2de6844599d3c913de06de313/guide/concepts/scaling-lucene/replication/index.textile](https://github.com/lukas-vlcek/elasticsearch.github.com/blob/cd92cb1cfaa238d2de6844599d3c913de06de313/guide/concepts/scaling-lucene/replication/index.textile)
3. To my knowledge, ES does not move shards around the cluster based on the
query or work load. But there are options how to setup and update shard
allocation even in real-time:
[Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/allocation.html)
[Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/modules/cluster.html)
So it should be possible to implement external system, that does this based
on the load of nodes (and the load can be check using ES admin API, that is
the API which bigdesk uses to create all the charts).
Regards,
Lukas
On Wed, Apr 4, 2012 at 9:49 PM, Oren Mazor [oren@wildbit.com](mailto:oren@wildbit.com) wrote:
> hey guys,
>
> I've been on/off dealing with some ES issues, and I think I've
> resolved a LOT of them. but there's still one thing that bugs me.
>
> My structure:
>
> 40gb index
> two nodes, ten shards, one replica.
> each node is a quad core 32gb of ram machine. min=max=16gb
> our data is extremely uniform, and extremely small. but comes in at a
> high rate of around 100 index actions per second.
> ES is at 18.6 (I know, we'll be updating this week)
>
> currently, all of the shards are marked as primary on node one.
>
> node 1:
>
> uptime is 9 hours.
> big desk shows 11399 garbage collection actions, and 2000 merges.
> cpu has been uniform at under 5% for pretty much ever.
> memory is at used=30gb, actual=14gb
>
> node 2:
>
> uptime is 3 hours
> bigdesk shows 104000 gc actions, and 800 merges.
> cpu has been uniform at 80-90% for pretty much forever.
> memory is at used=21gb, actual=13gb
>
> note:
>
> node 1 is also running a few python scripts (server density, mongo
> sync tool, geckoboard agent)
>
> my questions:
>
> - why is cpu so high on node 2?
> - I was thinking that the 'primary' shards are the ones that do the
> merging, and the 'slaves' do the receives of the complete index, but
> in both cases lucene indices have to do a merge. not so much a
> question as making sure I'm understanding everything.
> - my data is routed, which in theory should mean that some shards
> would be doing a lot more work than others. does ES account for 'work'
> and move shards from machine to machine? or does it just account for
> shard size?
>
> thanks!
> Oren
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [April 4, 2012, 9:35pm UTC](https://discuss.elastic.co/t/distribution-of-work/7231/3 "2012-04-04T21:35:58Z")
Because the number of replicas is 1 and you have 2 nodes, then it is
strange that one node has high CPU load compared to the other... . Is the
OS the same on both? Are you running on a virtualized / AWS env? Is the JVM
the same on both? Can you gist a couple of jstack results from the node
with the high load? (
[jstack - Stack Trace](http://docs.oracle.com/javase/1.5.0/docs/tooldocs/share/jstack.html)).
On Wed, Apr 4, 2012 at 10:49 PM, Oren Mazor [oren@wildbit.com](mailto:oren@wildbit.com) wrote:
> hey guys,
>
> I've been on/off dealing with some ES issues, and I think I've
> resolved a LOT of them. but there's still one thing that bugs me.
>
> My structure:
>
> 40gb index
> two nodes, ten shards, one replica.
> each node is a quad core 32gb of ram machine. min=max=16gb
> our data is extremely uniform, and extremely small. but comes in at a
> high rate of around 100 index actions per second.
> ES is at 18.6 (I know, we'll be updating this week)
>
> currently, all of the shards are marked as primary on node one.
>
> node 1:
>
> uptime is 9 hours.
> big desk shows 11399 garbage collection actions, and 2000 merges.
> cpu has been uniform at under 5% for pretty much ever.
> memory is at used=30gb, actual=14gb
>
> node 2:
>
> uptime is 3 hours
> bigdesk shows 104000 gc actions, and 800 merges.
> cpu has been uniform at 80-90% for pretty much forever.
> memory is at used=21gb, actual=13gb
>
> note:
>
> node 1 is also running a few python scripts (server density, mongo
> sync tool, geckoboard agent)
>
> my questions:
>
> - why is cpu so high on node 2?
> - I was thinking that the 'primary' shards are the ones that do the
> merging, and the 'slaves' do the receives of the complete index, but
> in both cases lucene indices have to do a merge. not so much a
> question as making sure I'm understanding everything.
> - my data is routed, which in theory should mean that some shards
> would be doing a lot more work than others. does ES account for 'work'
> and move shards from machine to machine? or does it just account for
> shard size?
>
> thanks!
> Oren
---
### Author:  [@orenmazor](https://discuss.elastic.co/u/orenmazor)
#### Post date: [April 5, 2012, 5:05am UTC](https://discuss.elastic.co/t/distribution-of-work/7231/4 "2012-04-05T05:05:32Z")
these are two RS machines, identical in every way other than the busy one
(node 2), is all non-primaries. all of the queries are going via node 1,
the one with the low cpu. node 2 (high cpu) is never actually queried,
which makes this even weirder.
if replication is basically just mirroring of requests, then the merge
activity should be identical on both, no?
jstack with no special flags:
>
On Wednesday, April 4, 2012 5:35:58 PM UTC-4, kimchy wrote:
> Because the number of replicas is 1 and you have 2 nodes, then it is
> strange that one node has high CPU load compared to the other... . Is the
> OS the same on both? Are you running on a virtualized / AWS env? Is the JVM
> the same on both? Can you gist a couple of jstack results from the node
> with the high load? (
> [jstack - Stack Trace](http://docs.oracle.com/javase/1.5.0/docs/tooldocs/share/jstack.html)).
>
> On Wed, Apr 4, 2012 at 10:49 PM, Oren Mazor [oren@wildbit.com](mailto:oren@wildbit.com) wrote:
>
> > hey guys,
> >
> > I've been on/off dealing with some ES issues, and I think I've
> > resolved a LOT of them. but there's still one thing that bugs me.
> >
> > My structure:
> >
> > 40gb index
> > two nodes, ten shards, one replica.
> > each node is a quad core 32gb of ram machine. min=max=16gb
> > our data is extremely uniform, and extremely small. but comes in at a
> > high rate of around 100 index actions per second.
> > ES is at 18.6 (I know, we'll be updating this week)
> >
> > currently, all of the shards are marked as primary on node one.
> >
> > node 1:
> >
> > uptime is 9 hours.
> > big desk shows 11399 garbage collection actions, and 2000 merges.
> > cpu has been uniform at under 5% for pretty much ever.
> > memory is at used=30gb, actual=14gb
> >
> > node 2:
> >
> > uptime is 3 hours
> > bigdesk shows 104000 gc actions, and 800 merges.
> > cpu has been uniform at 80-90% for pretty much forever.
> > memory is at used=21gb, actual=13gb
> >
> > note:
> >
> > node 1 is also running a few python scripts (server density, mongo
> > sync tool, geckoboard agent)
> >
> > my questions:
> >
> > - why is cpu so high on node 2?
> > - I was thinking that the 'primary' shards are the ones that do the
> > merging, and the 'slaves' do the receives of the complete index, but
> > in both cases lucene indices have to do a merge. not so much a
> > question as making sure I'm understanding everything.
> > - my data is routed, which in theory should mean that some shards
> > would be doing a lot more work than others. does ES account for 'work'
> > and move shards from machine to machine? or does it just account for
> > shard size?
> >
> > thanks!
> > Oren
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [April 5, 2012, 3:25pm UTC](https://discuss.elastic.co/t/distribution-of-work/7231/5 "2012-04-05T15:25:47Z")
On Thu, Apr 5, 2012 at 8:05 AM, Oren Mazor [oren@wildbit.com](mailto:oren@wildbit.com) wrote:
> these are two RS machines, identical in every way other than the busy one
> (node 2), is all non-primaries. all of the queries are going via node 1,
> the one with the low cpu. node 2 (high cpu) is never actually queried,
> which makes this even weirder.
Even if you don't send explicit search request to them, they are still
being searched on because elasticsearch will distribute the data between
them.
> if replication is basically just mirroring of requests, then the merge
> activity should be identical on both, no?
First, note that the merge stats are only from the time the node was
started. And no, even if both are started on teh same time, they will
differ because they are not completely in sync (doc wise they are, time
wise of when things kick off, like merges, might differ).
> jstack with no special flags:
> [gist:483d2cbc73c29056ac44 · GitHub](https://gist.github.com/483d2cbc73c29056ac44)
Can you gist another set of jstack? Can you also include one from node1?
> On Wednesday, April 4, 2012 5:35:58 PM UTC-4, kimchy wrote:
>
> > Because the number of replicas is 1 and you have 2 nodes, then it is
> > strange that one node has high CPU load compared to the other... . Is the
> > OS the same on both? Are you running on a virtualized / AWS env? Is the JVM
> > the same on both? Can you gist a couple of jstack results from the node
> > with the high load? ([Moved](http://docs.oracle.com/**javase/1.5.0/docs/tooldocs/)
> > \*\*share/jstack.html[http://docs.oracle.com/javase/1.5.0/docs/tooldocs/share/jstack.html](http://docs.oracle.com/javase/1.5.0/docs/tooldocs/share/jstack.html)
> > ).
> >
> > On Wed, Apr 4, 2012 at 10:49 PM, Oren Mazor [oren@wildbit.com](mailto:oren@wildbit.com) wrote:
> >
> > > hey guys,
> > >
> > > I've been on/off dealing with some ES issues, and I think I've
> > > resolved a LOT of them. but there's still one thing that bugs me.
> > >
> > > My structure:
> > >
> > > 40gb index
> > > two nodes, ten shards, one replica.
> > > each node is a quad core 32gb of ram machine. min=max=16gb
> > > our data is extremely uniform, and extremely small. but comes in at a
> > > high rate of around 100 index actions per second.
> > > ES is at 18.6 (I know, we'll be updating this week)
> > >
> > > currently, all of the shards are marked as primary on node one.
> > >
> > > node 1:
> > >
> > > uptime is 9 hours.
> > > big desk shows 11399 garbage collection actions, and 2000 merges.
> > > cpu has been uniform at under 5% for pretty much ever.
> > > memory is at used=30gb, actual=14gb
> > >
> > > node 2:
> > >
> > > uptime is 3 hours
> > > bigdesk shows 104000 gc actions, and 800 merges.
> > > cpu has been uniform at 80-90% for pretty much forever.
> > > memory is at used=21gb, actual=13gb
> > >
> > > note:
> > >
> > > node 1 is also running a few python scripts (server density, mongo
> > > sync tool, geckoboard agent)
> > >
> > > my questions:
> > >
> > > - why is cpu so high on node 2?
> > > - I was thinking that the 'primary' shards are the ones that do the
> > > merging, and the 'slaves' do the receives of the complete index, but
> > > in both cases lucene indices have to do a merge. not so much a
> > > question as making sure I'm understanding everything.
> > > - my data is routed, which in theory should mean that some shards
> > > would be doing a lot more work than others. does ES account for 'work'
> > > and move shards from machine to machine? or does it just account for
> > > shard size?
> > >
> > > thanks!
> > > Oren
---
### Author:  [@Oren\_Mazor](https://discuss.elastic.co/u/Oren_Mazor)
#### Post date: [April 17, 2012, 7:20pm UTC](https://discuss.elastic.co/t/distribution-of-work/7231/6 "2012-04-17T19:20:05Z")
so, a follow up:
we migrated to 19.2 and I'm seeing the same thing. the node that takes
the "point" of the requests has the cpu pegged again.
but with the new version of ES comes a new version of big desk (I
can't love this tool enough), and in this one I noticed that that node
has over 100k http connections, while the second node only 20. going
to look into balancing this out first.
On Apr 5, 11:25 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
> On Thu, Apr 5, 2012 at 8:05 AM, Oren Mazor [o...@wildbit.com](mailto:o...@wildbit.com) wrote:
>
> > these are two RS machines, identical in every way other than the busy one
> > (node 2), is all non-primaries. all of the queries are going via node 1,
> > the one with the low cpu. node 2 (high cpu) is never actually queried,
> > which makes this even weirder.
>
> Even if you don't send explicit search request to them, they are still
> being searched on because elasticsearch will distribute the data between
> them.
>
> > if replication is basically just mirroring of requests, then the merge
> > activity should be identical on both, no?
>
> First, note that the merge stats are only from the time the node was
> started. And no, even if both are started on teh same time, they will
> differ because they are not completely in sync (doc wise they are, time
> wise of when things kick off, like merges, might differ).
>
> > jstack with no special flags:
> > [gist:483d2cbc73c29056ac44 · GitHub](https://gist.github.com/483d2cbc73c29056ac44)
>
> Can you gist another set of jstack? Can you also include one from node1?
>
> > On Wednesday, April 4, 2012 5:35:58 PM UTC-4, kimchy wrote:
>
> > > Because the number of replicas is 1 and you have 2 nodes, then it is
> > > strange that one node has high CPU load compared to the other... . Is the
> > > OS the same on both? Are you running on a virtualized / AWS env? Is the JVM
> > > the same on both? Can you gist a couple of jstack results from the node
> > > with the high load? ([Moved](http://docs.oracle.com/**javase/1.5.0/docs/tooldocs/)
> > > \*\*share/jstack.html[http://docs.oracle.com/javase/1.5.0/docs/tooldocs/share/jstack.html](http://docs.oracle.com/javase/1.5.0/docs/tooldocs/share/jstack.html)
> > > ).
>
> > > On Wed, Apr 4, 2012 at 10:49 PM, Oren Mazor [o...@wildbit.com](mailto:o...@wildbit.com) wrote:
>
> > > > hey guys,
>
> > > > I've been on/off dealing with some ES issues, and I think I've
> > > > resolved a LOT of them. but there's still one thing that bugs me.
>
> > > > My structure:
>
> > > > 40gb index
> > > > two nodes, ten shards, one replica.
> > > > each node is a quad core 32gb of ram machine. min=max=16gb
> > > > our data is extremely uniform, and extremely small. but comes in at a
> > > > high rate of around 100 index actions per second.
> > > > ES is at 18.6 (I know, we'll be updating this week)
>
> > > > currently, all of the shards are marked as primary on node one.
>
> > > > node 1:
>
> > > > uptime is 9 hours.
> > > > big desk shows 11399 garbage collection actions, and 2000 merges.
> > > > cpu has been uniform at under 5% for pretty much ever.
> > > > memory is at used=30gb, actual=14gb
>
> > > > node 2:
>
> > > > uptime is 3 hours
> > > > bigdesk shows 104000 gc actions, and 800 merges.
> > > > cpu has been uniform at 80-90% for pretty much forever.
> > > > memory is at used=21gb, actual=13gb
>
> > > > note:
>
> > > > node 1 is also running a few python scripts (server density, mongo
> > > > sync tool, geckoboard agent)
>
> > > > my questions:
>
> > > > - why is cpu so high on node 2?
> > > > - I was thinking that the 'primary' shards are the ones that do the
> > > > merging, and the 'slaves' do the receives of the complete index, but
> > > > in both cases lucene indices have to do a merge. not so much a
> > > > question as making sure I'm understanding everything.
> > > > - my data is routed, which in theory should mean that some shards
> > > > would be doing a lot more work than others. does ES account for 'work'
> > > > and move shards from machine to machine? or does it just account for
> > > > shard size?
>
> > > > thanks!
> > > > Oren
---
### Author:  [@Oren\_Mazor](https://discuss.elastic.co/u/Oren_Mazor)
#### Post date: [April 21, 2012, 7:15pm UTC](https://discuss.elastic.co/t/distribution-of-work/7231/7 "2012-04-21T19:15:04Z")
I've noticed another thing. the nodes have different data. is this a
function of re-play of index/delete actions? is there a way to modify
the frequency that happens? it looks like its a day or two behind…
On Apr 17, 3:20 pm, Oren Mazor [oren.ma...@gmail.com](mailto:oren.ma...@gmail.com) wrote:
> so, a follow up:
>
> we migrated to 19.2 and I'm seeing the same thing. thenodethat takes
> the "point" of the requests has the cpu pegged again.
>
> but with the new version of ES comes a new version of big desk (I
> can't love this tool enough), and in this one I noticed that thatnode
> has over 100k http connections, while the secondnodeonly 20. going
> to look into balancing this out first.
>
> On Apr 5, 11:25 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
>
> > On Thu, Apr 5, 2012 at 8:05 AM, Oren Mazor [o...@wildbit.com](mailto:o...@wildbit.com) wrote:
> >
> > > these are two RS machines, identical in every way other than the busy one
> > > (node2), is all non-primaries. all of the queries are going vianode1,
> > > the one with the low cpu.node2 (high cpu) is never actually queried,
> > > which makes this even weirder.
>
> > Even if you don't send explicit search request to them, they are still
> > being searched on because elasticsearch will distribute the data between
> > them.
>
> > > if replication is basically just mirroring of requests, then the merge
> > > activity should be identical on both, no?
>
> > First, note that the merge stats are only from the time thenodewas
> > started. And no, even if both are started on teh same time, they will
> > differ because they are not completely insync(doc wise they are, time
> > wise of when things kick off, like merges, might differ).
>
> > > jstack with no special flags:
> > > [gist:483d2cbc73c29056ac44 · GitHub](https://gist.github.com/483d2cbc73c29056ac44)
>
> > Can you gist another set of jstack? Can you also include one from node1?
>
> > > On Wednesday, April 4, 2012 5:35:58 PM UTC-4, kimchy wrote:
>
> > > > Because the number of replicas is 1 and you have 2 nodes, then it is
> > > > strange that onenodehas high CPU load compared to the other... . Is the
> > > > OS the same on both? Are you running on a virtualized / AWS env? Is the JVM
> > > > the same on both? Can you gist a couple of jstack results from thenode
> > > > with the high load? ([Moved](http://docs.oracle.com/**javase/1.5.0/docs/tooldocs/)
> > > > \*\*share/jstack.html[http://docs.oracle.com/javase/1.5.0/docs/tooldocs/share/jstack.html](http://docs.oracle.com/javase/1.5.0/docs/tooldocs/share/jstack.html)
> > > > ).
>
> > > > On Wed, Apr 4, 2012 at 10:49 PM, Oren Mazor [o...@wildbit.com](mailto:o...@wildbit.com) wrote:
>
> > > > > hey guys,
>
> > > > > I've been on/off dealing with some ES issues, and I think I've
> > > > > resolved a LOT of them. but there's still one thing that bugs me.
>
> > > > > My structure:
>
> > > > > 40gb index
> > > > > two nodes, ten shards, one replica.
> > > > > eachnodeis a quad core 32gb of ram machine. min=max=16gb
> > > > > our data is extremely uniform, and extremely small. but comes in at a
> > > > > high rate of around 100 index actions per second.
> > > > > ES is at 18.6 (I know, we'll be updating this week)
>
> > > > > currently, all of the shards are marked as primary onnodeone.
>
> > > > > node1:
>
> > > > > uptime is 9 hours.
> > > > > big desk shows 11399 garbage collection actions, and 2000 merges.
> > > > > cpu has been uniform at under 5% for pretty much ever.
> > > > > memory is at used=30gb, actual=14gb
>
> > > > > node2:
>
> > > > > uptime is 3 hours
> > > > > bigdesk shows 104000 gc actions, and 800 merges.
> > > > > cpu has been uniform at 80-90% for pretty much forever.
> > > > > memory is at used=21gb, actual=13gb
>
> > > > > note:
>
> > > > > node1 is also running a few python scripts (server density, mongo
> > > > > synctool, geckoboard agent)
>
> > > > > my questions:
>
> > > > > - why is cpu so high onnode2?
> > > > > - I was thinking that the 'primary' shards are the ones that do the
> > > > > merging, and the 'slaves' do the receives of the complete index, but
> > > > > in both cases lucene indices have to do a merge. not so much a
> > > > > question as making sure I'm understanding everything.
> > > > > - my data is routed, which in theory should mean that some shards
> > > > > would be doing a lot more work than others. does ES account for 'work'
> > > > > and move shards from machine to machine? or does it just account for
> > > > > shard size?
>
> > > > > thanks!
> > > > > Oren
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [April 25, 2012, 3:31pm UTC](https://discuss.elastic.co/t/distribution-of-work/7231/8 "2012-04-25T15:31:32Z")
What do you mean by a day or two behind?
On Sat, Apr 21, 2012 at 10:15 PM, Oren Mazor [oren.mazor@gmail.com](mailto:oren.mazor@gmail.com) wrote:
> I've noticed another thing. the nodes have different data. is this a
> function of re-play of index/delete actions? is there a way to modify
> the frequency that happens? it looks like its a day or two behind…
>
> On Apr 17, 3:20 pm, Oren Mazor [oren.ma...@gmail.com](mailto:oren.ma...@gmail.com) wrote:
>
> > so, a follow up:
> >
> > we migrated to 19.2 and I'm seeing the same thing. thenodethat takes
> > the "point" of the requests has the cpu pegged again.
> >
> > but with the new version of ES comes a new version of big desk (I
> > can't love this tool enough), and in this one I noticed that thatnode
> > has over 100k http connections, while the secondnodeonly 20. going
> > to look into balancing this out first.
> >
> > On Apr 5, 11:25 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
> >
> > > On Thu, Apr 5, 2012 at 8:05 AM, Oren Mazor [o...@wildbit.com](mailto:o...@wildbit.com) wrote:
> > >
> > > > these are two RS machines, identical in every way other than the
> > > > busy one
> > > > (node2), is all non-primaries. all of the queries are going vianode1,
> > > > the one with the low cpu.node2 (high cpu) is never actually queried,
> > > > which makes this even weirder.
> >
> > > Even if you don't send explicit search request to them, they are still
> > > being searched on because elasticsearch will distribute the data
> > > between
> > > them.
> >
> > > > if replication is basically just mirroring of requests, then the
> > > > merge
> > > > activity should be identical on both, no?
> >
> > > First, note that the merge stats are only from the time thenodewas
> > > started. And no, even if both are started on teh same time, they will
> > > differ because they are not completely insync(doc wise they are, time
> > > wise of when things kick off, like merges, might differ).
> >
> > > > jstack with no special flags:
> > > > [gist:483d2cbc73c29056ac44 · GitHub](https://gist.github.com/483d2cbc73c29056ac44)
> >
> > > Can you gist another set of jstack? Can you also include one from
> > > node1?
> >
> > > > On Wednesday, April 4, 2012 5:35:58 PM UTC-4, kimchy wrote:
> >
> > > > > Because the number of replicas is 1 and you have 2 nodes, then it is
> > > > > strange that onenodehas high CPU load compared to the other... . Is
> > > > > the
> > > > > OS the same on both? Are you running on a virtualized / AWS env? Is
> > > > > the JVM
> > > > > the same on both? Can you gist a couple of jstack results from
> > > > > thenode
> > > > > with the high load? (
> > > > > [Moved](http://docs.oracle.com/**javase/1.5.0/docs/tooldocs/)
> > > > > \*\*share/jstack.html\<
> > > > > [jstack - Stack Trace](http://docs.oracle.com/javase/1.5.0/docs/tooldocs/share/jstack.html)\>
> > > > > ).
> >
> > > > > On Wed, Apr 4, 2012 at 10:49 PM, Oren Mazor [o...@wildbit.com](mailto:o...@wildbit.com)
> > > > > wrote:
> >
> > > > > > hey guys,
> >
> > > > > > I've been on/off dealing with some ES issues, and I think I've
> > > > > > resolved a LOT of them. but there's still one thing that bugs me.
> >
> > > > > > My structure:
> >
> > > > > > 40gb index
> > > > > > two nodes, ten shards, one replica.
> > > > > > eachnodeis a quad core 32gb of ram machine. min=max=16gb
> > > > > > our data is extremely uniform, and extremely small. but comes in
> > > > > > at a
> > > > > > high rate of around 100 index actions per second.
> > > > > > ES is at 18.6 (I know, we'll be updating this week)
> >
> > > > > > currently, all of the shards are marked as primary onnodeone.
> >
> > > > > > node1:
> >
> > > > > > uptime is 9 hours.
> > > > > > big desk shows 11399 garbage collection actions, and 2000 merges.
> > > > > > cpu has been uniform at under 5% for pretty much ever.
> > > > > > memory is at used=30gb, actual=14gb
> >
> > > > > > node2:
> >
> > > > > > uptime is 3 hours
> > > > > > bigdesk shows 104000 gc actions, and 800 merges.
> > > > > > cpu has been uniform at 80-90% for pretty much forever.
> > > > > > memory is at used=21gb, actual=13gb
> >
> > > > > > note:
> >
> > > > > > node1 is also running a few python scripts (server density, mongo
> > > > > > synctool, geckoboard agent)
> >
> > > > > > my questions:
> >
> > > > > > - why is cpu so high onnode2?
> > > > > > - I was thinking that the 'primary' shards are the ones that do
> > > > > > the
> > > > > > merging, and the 'slaves' do the receives of the complete index,
> > > > > > but
> > > > > > in both cases lucene indices have to do a merge. not so much a
> > > > > > question as making sure I'm understanding everything.
> > > > > > - my data is routed, which in theory should mean that some shards
> > > > > > would be doing a lot more work than others. does ES account for
> > > > > > 'work'
> > > > > > and move shards from machine to machine? or does it just account
> > > > > > for
> > > > > > shard size?
> >
> > > > > > thanks!
> > > > > > Oren
---
### Author:  [@Oren\_Mazor](https://discuss.elastic.co/u/Oren_Mazor)
#### Post date: [April 27, 2012, 3:54pm UTC](https://discuss.elastic.co/t/distribution-of-work/7231/9 "2012-04-27T15:54:30Z")
Lets say I index a new document every minute. with replica turned on,
it means that one of my nodes is going to be missing an entire day's
worth of indexing activity.
I'm currently running without replicas (but with multiple nodes) and
its running great. as soon as I turn on replica, I can get two
different results for a search query.
let me know what debug info I can get for you on this.
On Apr 25, 11:31 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
> What do you mean by a day or two behind?
>
> On Sat, Apr 21, 2012 at 10:15 PM, Oren Mazor [oren.ma...@gmail.com](mailto:oren.ma...@gmail.com) wrote:
>
> > I've noticed another thing. the nodes have different data. is this a
> > function of re-play of index/delete actions? is there a way to modify
> > the frequency that happens? it looks like its a day or two behind…
>
> > On Apr 17, 3:20 pm, Oren Mazor [oren.ma...@gmail.com](mailto:oren.ma...@gmail.com) wrote:
> >
> > > so, a follow up:
>
> > > we migrated to 19.2 and I'm seeing the same thing. thenodethat takes
> > > the "point" of the requests has the cpu pegged again.
>
> > > but with the new version of ES comes a new version of big desk (I
> > > can't love this tool enough), and in this one I noticed that thatnode
> > > has over 100k http connections, while the secondnodeonly 20. going
> > > to look into balancing this out first.
>
> > > On Apr 5, 11:25 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
>
> > > > On Thu, Apr 5, 2012 at 8:05 AM, Oren Mazor [o...@wildbit.com](mailto:o...@wildbit.com) wrote:
> > > >
> > > > > these are two RS machines, identical in every way other than the
> > > > > busy one
> > > > > (node2), is all non-primaries. all of the queries are going vianode1,
> > > > > the one with the low cpu.node2 (high cpu) is never actually queried,
> > > > > which makes this even weirder.
>
> > > > Even if you don't send explicit search request to them, they are still
> > > > being searched on because elasticsearch will distribute the data
> > > > between
> > > > them.
>
> > > > > if replication is basically just mirroring of requests, then the
> > > > > merge
> > > > > activity should be identical on both, no?
>
> > > > First, note that the merge stats are only from the time thenodewas
> > > > started. And no, even if both are started on teh same time, they will
> > > > differ because they are not completely insync(doc wise they are, time
> > > > wise of when things kick off, like merges, might differ).
>
> > > > > jstack with no special flags:
> > > > > [gist:483d2cbc73c29056ac44 · GitHub](https://gist.github.com/483d2cbc73c29056ac44)
>
> > > > Can you gist another set of jstack? Can you also include one from
> > > > node1?
>
> > > > > On Wednesday, April 4, 2012 5:35:58 PM UTC-4, kimchy wrote:
>
> > > > > > Because the number of replicas is 1 and you have 2 nodes, then it is
> > > > > > strange that onenodehas high CPU load compared to the other... . Is
> > > > > > the
> > > > > > OS the same on both? Are you running on a virtualized / AWS env? Is
> > > > > > the JVM
> > > > > > the same on both? Can you gist a couple of jstack results from
> > > > > > thenode
> > > > > > with the high load? (
> > > > > > [Moved](http://docs.oracle.com/**javase/1.5.0/docs/tooldocs/)
> > > > > > \*\*share/jstack.html\<
> > > > > > [jstack - Stack Trace](http://docs.oracle.com/javase/1.5.0/docs/tooldocs/share/jstack.html)\>
> > > > > > ).
>
> > > > > > On Wed, Apr 4, 2012 at 10:49 PM, Oren Mazor [o...@wildbit.com](mailto:o...@wildbit.com)
> > > > > > wrote:
>
> > > > > > > hey guys,
>
> > > > > > > I've been on/off dealing with some ES issues, and I think I've
> > > > > > > resolved a LOT of them. but there's still one thing that bugs me.
>
> > > > > > > My structure:
>
> > > > > > > 40gb index
> > > > > > > two nodes, ten shards, one replica.
> > > > > > > eachnodeis a quad core 32gb of ram machine. min=max=16gb
> > > > > > > our data is extremely uniform, and extremely small. but comes in
> > > > > > > at a
> > > > > > > high rate of around 100 index actions per second.
> > > > > > > ES is at 18.6 (I know, we'll be updating this week)
>
> > > > > > > currently, all of the shards are marked as primary onnodeone.
>
> > > > > > > node1:
>
> > > > > > > uptime is 9 hours.
> > > > > > > big desk shows 11399 garbage collection actions, and 2000 merges.
> > > > > > > cpu has been uniform at under 5% for pretty much ever.
> > > > > > > memory is at used=30gb, actual=14gb
>
> > > > > > > node2:
>
> > > > > > > uptime is 3 hours
> > > > > > > bigdesk shows 104000 gc actions, and 800 merges.
> > > > > > > cpu has been uniform at 80-90% for pretty much forever.
> > > > > > > memory is at used=21gb, actual=13gb
>
> > > > > > > note:
>
> > > > > > > node1 is also running a few python scripts (server density, mongo
> > > > > > > synctool, geckoboard agent)
>
> > > > > > > my questions:
>
> > > > > > > - why is cpu so high onnode2?
> > > > > > > - I was thinking that the 'primary' shards are the ones that do
> > > > > > > the
> > > > > > > merging, and the 'slaves' do the receives of the complete index,
> > > > > > > but
> > > > > > > in both cases lucene indices have to do a merge. not so much a
> > > > > > > question as making sure I'm understanding everything.
> > > > > > > - my data is routed, which in theory should mean that some shards
> > > > > > > would be doing a lot more work than others. does ES account for
> > > > > > > 'work'
> > > > > > > and move shards from machine to machine? or does it just account
> > > > > > > for
> > > > > > > shard size?
>
> > > > > > > thanks!
> > > > > > > Oren
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [April 29, 2012, 4:51pm UTC](https://discuss.elastic.co/t/distribution-of-work/7231/10 "2012-04-29T16:51:32Z")
Replication is synchronous, once you index a document, it will also be
indexed on the replica. Are you playing with the refresh interval by any
chance?
On Fri, Apr 27, 2012 at 6:54 PM, Oren Mazor [oren.mazor@gmail.com](mailto:oren.mazor@gmail.com) wrote:
> Lets say I index a new document every minute. with replica turned on,
> it means that one of my nodes is going to be missing an entire day's
> worth of indexing activity.
>
> I'm currently running without replicas (but with multiple nodes) and
> its running great. as soon as I turn on replica, I can get two
> different results for a search query.
>
> let me know what debug info I can get for you on this.
>
> On Apr 25, 11:31 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
>
> > What do you mean by a day or two behind?
> >
> > On Sat, Apr 21, 2012 at 10:15 PM, Oren Mazor [oren.ma...@gmail.com](mailto:oren.ma...@gmail.com)
> > wrote:
> >
> > > I've noticed another thing. the nodes have different data. is this a
> > > function of re-play of index/delete actions? is there a way to modify
> > > the frequency that happens? it looks like its a day or two behind…
> >
> > > On Apr 17, 3:20 pm, Oren Mazor [oren.ma...@gmail.com](mailto:oren.ma...@gmail.com) wrote:
> > >
> > > > so, a follow up:
> >
> > > > we migrated to 19.2 and I'm seeing the same thing. thenodethat takes
> > > > the "point" of the requests has the cpu pegged again.
> >
> > > > but with the new version of ES comes a new version of big desk (I
> > > > can't love this tool enough), and in this one I noticed that thatnode
> > > > has over 100k http connections, while the secondnodeonly 20. going
> > > > to look into balancing this out first.
> >
> > > > On Apr 5, 11:25 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
> >
> > > > > On Thu, Apr 5, 2012 at 8:05 AM, Oren Mazor [o...@wildbit.com](mailto:o...@wildbit.com)
> > > > > wrote:
> > > > >
> > > > > > these are two RS machines, identical in every way other than the
> > > > > > busy one
> > > > > > (node2), is all non-primaries. all of the queries are going
> > > > > > vianode1,
> > > > > > the one with the low cpu.node2 (high cpu) is never actually
> > > > > > queried,
> > > > > > which makes this even weirder.
> >
> > > > > Even if you don't send explicit search request to them, they are
> > > > > still
> > > > > being searched on because elasticsearch will distribute the data
> > > > > between
> > > > > them.
> >
> > > > > > if replication is basically just mirroring of requests, then the
> > > > > > merge
> > > > > > activity should be identical on both, no?
> >
> > > > > First, note that the merge stats are only from the time thenodewas
> > > > > started. And no, even if both are started on teh same time, they
> > > > > will
> > > > > differ because they are not completely insync(doc wise they are,
> > > > > time
> > > > > wise of when things kick off, like merges, might differ).
> >
> > > > > > jstack with no special flags:
> > > > > > [gist:483d2cbc73c29056ac44 · GitHub](https://gist.github.com/483d2cbc73c29056ac44)
> >
> > > > > Can you gist another set of jstack? Can you also include one from
> > > > > node1?
> >
> > > > > > On Wednesday, April 4, 2012 5:35:58 PM UTC-4, kimchy wrote:
> >
> > > > > > > Because the number of replicas is 1 and you have 2 nodes, then
> > > > > > > it is
> > > > > > > strange that onenodehas high CPU load compared to the other...
> > > > > > > . Is
> > > > > > > the
> > > > > > > OS the same on both? Are you running on a virtualized / AWS
> > > > > > > env? Is
> > > > > > > the JVM
> > > > > > > the same on both? Can you gist a couple of jstack results from
> > > > > > > thenode
> > > > > > > with the high load? (
> > > > > > > [Moved](http://docs.oracle.com/**javase/1.5.0/docs/tooldocs/)
> > > > > > > \*\*share/jstack.html\<
> > > > > > > [jstack - Stack Trace](http://docs.oracle.com/javase/1.5.0/docs/tooldocs/share/jstack.html)\>
> > > > > > > ).
> >
> > > > > > > On Wed, Apr 4, 2012 at 10:49 PM, Oren Mazor [o...@wildbit.com](mailto:o...@wildbit.com)
> > > > > > > wrote:
> >
> > > > > > > > hey guys,
> >
> > > > > > > > I've been on/off dealing with some ES issues, and I think I've
> > > > > > > > resolved a LOT of them. but there's still one thing that bugs
> > > > > > > > me.
> >
> > > > > > > > My structure:
> >
> > > > > > > > 40gb index
> > > > > > > > two nodes, ten shards, one replica.
> > > > > > > > eachnodeis a quad core 32gb of ram machine. min=max=16gb
> > > > > > > > our data is extremely uniform, and extremely small. but comes
> > > > > > > > in
> > > > > > > > at a
> > > > > > > > high rate of around 100 index actions per second.
> > > > > > > > ES is at 18.6 (I know, we'll be updating this week)
> >
> > > > > > > > currently, all of the shards are marked as primary onnodeone.
> >
> > > > > > > > node1:
> >
> > > > > > > > uptime is 9 hours.
> > > > > > > > big desk shows 11399 garbage collection actions, and 2000
> > > > > > > > merges.
> > > > > > > > cpu has been uniform at under 5% for pretty much ever.
> > > > > > > > memory is at used=30gb, actual=14gb
> >
> > > > > > > > node2:
> >
> > > > > > > > uptime is 3 hours
> > > > > > > > bigdesk shows 104000 gc actions, and 800 merges.
> > > > > > > > cpu has been uniform at 80-90% for pretty much forever.
> > > > > > > > memory is at used=21gb, actual=13gb
> >
> > > > > > > > note:
> >
> > > > > > > > node1 is also running a few python scripts (server density,
> > > > > > > > mongo
> > > > > > > > synctool, geckoboard agent)
> >
> > > > > > > > my questions:
> >
> > > > > > > > - why is cpu so high onnode2?
> > > > > > > > - I was thinking that the 'primary' shards are the ones that
> > > > > > > > do
> > > > > > > > the
> > > > > > > > merging, and the 'slaves' do the receives of the complete
> > > > > > > > index,
> > > > > > > > but
> > > > > > > > in both cases lucene indices have to do a merge. not so much a
> > > > > > > > question as making sure I'm understanding everything.
> > > > > > > > - my data is routed, which in theory should mean that some
> > > > > > > > shards
> > > > > > > > would be doing a lot more work than others. does ES account for
> > > > > > > > 'work'
> > > > > > > > and move shards from machine to machine? or does it just
> > > > > > > > account
> > > > > > > > for
> > > > > > > > shard size?
> >
> > > > > > > > thanks!
> > > > > > > > Oren
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 3:30am UTC](https://discuss.elastic.co/t/distribution-of-work/7231/11 "2017-07-06T03:30:39Z")