Cluster state is compressed by default, but for large clusters, or those
with lots of large mappings, it can also be a problem.
The cluster needs to know about what shards make up an index, as well as
where they are located which is why. As you mentioned it is currently
stored under two separate areas of the cluster state, though it's possible
this could be combined to reduce the size.
I did some more digging here to understand things a bit more than my last
(lame, sorry!) email.
We only send routing_table over the network and then we build routing_nodes
out of it on the other side. However routing_nodes is actually built only
on access, so unless you use it, which happens on master nodes and if you
do get /_cluster/status, it's not actually built.
We need routing_nodes for a quick access to node view of the shard
allocation and we need routing_table for the index view of shard
allocation. This is done to ensure performance.
There is work happening to improve the overall transfer speed of the
cluster state between nodes, essentially to send the delta of a state to
all nodes, rather than the whole state as currently happens.
Cluster state is compressed by default, but for large clusters, or those
with lots of large mappings, it can also be a problem.
The cluster needs to know about what shards make up an index, as well as
where they are located which is why. As you mentioned it is currently
stored under two separate areas of the cluster state, though it's possible
this could be combined to reduce the size.
All ok!
Thanks for the explanation! I just found it odd that the same information
was displayed twice, but if it's only sent one way that is good. Makes
sense!
Elasticsearch has already become incredibly more stable in the past 12
months!
The future of ES is really exciting!
Thanks again!
Rob
On Wednesday, March 25, 2015 at 11:22:14 PM UTC+1, Mark Walkom wrote:
I did some more digging here to understand things a bit more than my last
(lame, sorry!) email.
We only send routing_table over the network and then we build
routing_nodes out of it on the other side. However routing_nodes is
actually built only on access, so unless you use it, which happens on
master nodes and if you do get /_cluster/status, it's not actually built.
We need routing_nodes for a quick access to node view of the shard
allocation and we need routing_table for the index view of shard
allocation. This is done to ensure performance.
There is work happening to improve the overall transfer speed of the
cluster state between nodes, essentially to send the delta of a state to
all nodes, rather than the whole state as currently happens.
On 26 March 2015 at 09:02, Mark Walkom <markw...@gmail.com <javascript:>>
wrote:
Cluster state is compressed by default, but for large clusters, or those
with lots of large mappings, it can also be a problem.
The cluster needs to know about what shards make up an index, as well as
where they are located which is why. As you mentioned it is currently
stored under two separate areas of the cluster state, though it's possible
this could be combined to reduce the size.
On 25 March 2015 at 04:36, Robert Gardam <robert...@fyber.com
<javascript:>> wrote:
Hi,
I am starting to look at the size of my cluster state and I started to
notice that the shard information is duplicated.
One grouping seems to be from the view of the index and the other from
which shards live on which host.
I'm sure there's a logical reason for this, i'm just interested to know
why?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.