I have a cluster from 5 nodes, where I store the information from the
logstash.
Recently I've tried to increase the number of shards in the logstash index
to 20 (from 5).
From the beginning everything went well, all the shards were allocated and
the cluster state was green.
But, currently, when the new index is started (at the beginning of the
day), I met the situation when some shards are not allocated:
Nothing happens during the day, while I thing the cluster has resources.
If I would restart some nodes from the cluster, it could turn out that all
the shards would be allocated.
The idea which I'm trying to reach - more shards => each shard is smaller
=> it would allocate them faster, it would index faster, because of the
indexing on each shard.
Questions:
Why it is not able to allocate those shards during the index creation?
Why it does not allocate those shards during the day?
What is the recommended shards number for N nodes?
On Monday, November 17, 2014 11:02:43 AM UTC+2, Pavel P wrote:
Hi,
I have a cluster from 5 nodes, where I store the information from the
logstash.
Recently I've tried to increase the number of shards in the logstash index
to 20 (from 5).
From the beginning everything went well, all the shards were allocated and
the cluster state was green.
But, currently, when the new index is started (at the beginning of the
day), I met the situation when some shards are not allocated:
Nothing happens during the day, while I thing the cluster has resources.
If I would restart some nodes from the cluster, it could turn out that all
the shards would be allocated.
The idea which I'm trying to reach - more shards => each shard is smaller
=> it would allocate them faster, it would index faster, because of the
indexing on each shard.
Questions:
Why it is not able to allocate those shards during the index creation?
Why it does not allocate those shards during the day?
What is the recommended shards number for N nodes?
Your logic around having so many shards may make sense, but you still end
up having to move the same amount of data and it's more over head.
You should have N number of shards, where N is the number of nodes.
Check your logs, there may be something in there. Otherwise can you provide
more details on your node size; RAM, disk, CPU, ES and java version, that
sort of thing,
On Monday, November 17, 2014 11:02:43 AM UTC+2, Pavel P wrote:
Hi,
I have a cluster from 5 nodes, where I store the information from the
logstash.
Recently I've tried to increase the number of shards in the logstash
index to 20 (from 5).
From the beginning everything went well, all the shards were allocated
and the cluster state was green.
But, currently, when the new index is started (at the beginning of the
day), I met the situation when some shards are not allocated:
Nothing happens during the day, while I thing the cluster has resources.
If I would restart some nodes from the cluster, it could turn out that
all the shards would be allocated.
The idea which I'm trying to reach - more shards => each shard is smaller
=> it would allocate them faster, it would index faster, because of the
indexing on each shard.
Questions:
Why it is not able to allocate those shards during the index creation?
Why it does not allocate those shards during the day?
What is the recommended shards number for N nodes?
Unfortunately there is nothing in the logs about the issue.
Here is the hardware:
[image: Inline image 1]
The ES version is 1.4.0 , when I reduce the number of replicas to 0, it
would became green.
Here
the next is stated:
If a shard cannot be assigned, for example you’ve overallocated the number
of replicas for the number of nodes in the cluster, they will remain
UNASSIGNED.
How can I understand the real reason of unassigning the shards?
According to the bigdesk - the nodes are feeling themselves very good,
there are no queues and enough of the resources.
[image: Inline image 2]
I think that my hardware can hold more then 5 nodes. I've tried once to
reduce them to 10, but during the index creation some were still unassigned.
However, when I do restart of some node - it could be that every would be
assigned after the shards relocation process.
Your logic around having so many shards may make sense, but you still end
up having to move the same amount of data and it's more over head.
You should have N number of shards, where N is the number of nodes.
Check your logs, there may be something in there. Otherwise can you
provide more details on your node size; RAM, disk, CPU, ES and java
version, that sort of thing,
On Monday, November 17, 2014 11:02:43 AM UTC+2, Pavel P wrote:
Hi,
I have a cluster from 5 nodes, where I store the information from the
logstash.
Recently I've tried to increase the number of shards in the logstash
index to 20 (from 5).
From the beginning everything went well, all the shards were allocated
and the cluster state was green.
But, currently, when the new index is started (at the beginning of the
day), I met the situation when some shards are not allocated:
Nothing happens during the day, while I thing the cluster has resources.
If I would restart some nodes from the cluster, it could turn out that
all the shards would be allocated.
The idea which I'm trying to reach - more shards => each shard is
smaller => it would allocate them faster, it would index faster, because of
the indexing on each shard.
Questions:
Why it is not able to allocate those shards during the index creation?
Why it does not allocate those shards during the day?
What is the recommended shards number for N nodes?
This e-mail contains confidential and/or legally protected information. If
you are not the intended recipient or if you have received this e-mail by
error please notify the sender immediately and destroy this e-mail. Any
unauthorized review, copying, disclosure or distribution of the material in
this e-mail is strictly forbidden. The contents of this e-mail is legally
binding only if it is confirmed by letter or fax. The sending of e-mails to
us does not have any period-protecting effect. Thank you for your
cooperation.
And I think I have them migrating here and there, and while they are trying
to find the new home for themselves, it's not possible to allocate those 2
shards.
I've tried to close the index and then open - when the index was closed -
all the remaining shards were allocated successfully, after I've opened it
again - no shards remain unassigned.
So the current question is - why some shards are relocating constantly.
Regards,
On Monday, November 17, 2014 11:02:43 AM UTC+2, Pavel P wrote:
Hi,
I have a cluster from 5 nodes, where I store the information from the
logstash.
Recently I've tried to increase the number of shards in the logstash index
to 20 (from 5).
From the beginning everything went well, all the shards were allocated and
the cluster state was green.
But, currently, when the new index is started (at the beginning of the
day), I met the situation when some shards are not allocated:
Nothing happens during the day, while I thing the cluster has resources.
If I would restart some nodes from the cluster, it could turn out that all
the shards would be allocated.
The idea which I'm trying to reach - more shards => each shard is smaller
=> it would allocate them faster, it would index faster, because of the
indexing on each shard.
Questions:
Why it is not able to allocate those shards during the index creation?
Why it does not allocate those shards during the day?
What is the recommended shards number for N nodes?
And I think I have them migrating here and there, and while they are
trying to find the new home for themselves, it's not possible to allocate
those 2 shards.
I've tried to close the index and then open - when the index was closed -
all the remaining shards were allocated successfully, after I've opened it
again - no shards remain unassigned.
So the current question is - why some shards are relocating constantly.
Regards,
On Monday, November 17, 2014 11:02:43 AM UTC+2, Pavel P wrote:
Hi,
I have a cluster from 5 nodes, where I store the information from the
logstash.
Recently I've tried to increase the number of shards in the logstash
index to 20 (from 5).
From the beginning everything went well, all the shards were allocated
and the cluster state was green.
But, currently, when the new index is started (at the beginning of the
day), I met the situation when some shards are not allocated:
Nothing happens during the day, while I thing the cluster has resources.
If I would restart some nodes from the cluster, it could turn out that
all the shards would be allocated.
The idea which I'm trying to reach - more shards => each shard is smaller
=> it would allocate them faster, it would index faster, because of the
indexing on each shard.
Questions:
Why it is not able to allocate those shards during the index creation?
Why it does not allocate those shards during the day?
What is the recommended shards number for N nodes?
I've seen similar behaviour when i've had checksum some errors.There may be
some info in your logs about it. In that case shard fail to be assigned
because elasticsearch couldn't verify checksum of one or more of its
segments.
On Tuesday, 18 November 2014 14:17:02 UTC+1, Pavel P wrote:
It looks, that I have 2 shards unassigned, because two other shards were
stuck in "RELOCATING" state.
And I think I have them migrating here and there, and while they are
trying to find the new home for themselves, it's not possible to allocate
those 2 shards.
I've tried to close the index and then open - when the index was closed -
all the remaining shards were allocated successfully, after I've opened it
again - no shards remain unassigned.
So the current question is - why some shards are relocating constantly.
Regards,
On Monday, November 17, 2014 11:02:43 AM UTC+2, Pavel P wrote:
Hi,
I have a cluster from 5 nodes, where I store the information from the
logstash.
Recently I've tried to increase the number of shards in the logstash
index to 20 (from 5).
From the beginning everything went well, all the shards were allocated
and the cluster state was green.
But, currently, when the new index is started (at the beginning of the
day), I met the situation when some shards are not allocated:
Nothing happens during the day, while I thing the cluster has resources.
If I would restart some nodes from the cluster, it could turn out that
all the shards would be allocated.
The idea which I'm trying to reach - more shards => each shard is smaller
=> it would allocate them faster, it would index faster, because of the
indexing on each shard.
Questions:
Why it is not able to allocate those shards during the index creation?
Why it does not allocate those shards during the day?
What is the recommended shards number for N nodes?
I've seen similar behaviour when i've had checksum some errors.There may
be some info in your logs about it. In that case shard fail to be assigned
because elasticsearch couldn't verify checksum of one or more of its
segments.
On Tuesday, 18 November 2014 14:17:02 UTC+1, Pavel P wrote:
It looks, that I have 2 shards unassigned, because two other shards were
stuck in "RELOCATING" state.
And I think I have them migrating here and there, and while they are
trying to find the new home for themselves, it's not possible to allocate
those 2 shards.
I've tried to close the index and then open - when the index was closed -
all the remaining shards were allocated successfully, after I've opened it
again - no shards remain unassigned.
So the current question is - why some shards are relocating constantly.
Regards,
On Monday, November 17, 2014 11:02:43 AM UTC+2, Pavel P wrote:
Hi,
I have a cluster from 5 nodes, where I store the information from the
logstash.
Recently I've tried to increase the number of shards in the logstash
index to 20 (from 5).
From the beginning everything went well, all the shards were allocated
and the cluster state was green.
But, currently, when the new index is started (at the beginning of the
day), I met the situation when some shards are not allocated:
Nothing happens during the day, while I thing the cluster has resources.
If I would restart some nodes from the cluster, it could turn out that
all the shards would be allocated.
The idea which I'm trying to reach - more shards => each shard is
smaller => it would allocate them faster, it would index faster, because of
the indexing on each shard.
Questions:
Why it is not able to allocate those shards during the index creation?
Why it does not allocate those shards during the day?
What is the recommended shards number for N nodes?
This e-mail contains confidential and/or legally protected information. If
you are not the intended recipient or if you have received this e-mail by
error please notify the sender immediately and destroy this e-mail. Any
unauthorized review, copying, disclosure or distribution of the material in
this e-mail is strictly forbidden. The contents of this e-mail is legally
binding only if it is confirmed by letter or fax. The sending of e-mails to
us does not have any period-protecting effect. Thank you for your
cooperation.
I've seen similar behaviour when i've had checksum some errors.There may
be some info in your logs about it. In that case shard fail to be assigned
because elasticsearch couldn't verify checksum of one or more of its
segments.
On Tuesday, 18 November 2014 14:17:02 UTC+1, Pavel P wrote:
It looks, that I have 2 shards unassigned, because two other shards were
stuck in "RELOCATING" state.
And I think I have them migrating here and there, and while they are
trying to find the new home for themselves, it's not possible to allocate
those 2 shards.
I've tried to close the index and then open - when the index was closed
all the remaining shards were allocated successfully, after I've opened
it again - no shards remain unassigned.
So the current question is - why some shards are relocating constantly.
Regards,
On Monday, November 17, 2014 11:02:43 AM UTC+2, Pavel P wrote:
Hi,
I have a cluster from 5 nodes, where I store the information from the
logstash.
Recently I've tried to increase the number of shards in the logstash
index to 20 (from 5).
From the beginning everything went well, all the shards were allocated
and the cluster state was green.
But, currently, when the new index is started (at the beginning of the
day), I met the situation when some shards are not allocated:
Nothing happens during the day, while I thing the cluster has
resources.
If I would restart some nodes from the cluster, it could turn out that
all the shards would be allocated.
The idea which I'm trying to reach - more shards => each shard is
smaller => it would allocate them faster, it would index faster, because of
the indexing on each shard.
Questions:
Why it is not able to allocate those shards during the index
creation?
Why it does not allocate those shards during the day?
What is the recommended shards number for N nodes?
This e-mail contains confidential and/or legally protected information. If
you are not the intended recipient or if you have received this e-mail by
error please notify the sender immediately and destroy this e-mail. Any
unauthorized review, copying, disclosure or distribution of the material in
this e-mail is strictly forbidden. The contents of this e-mail is legally
binding only if it is confirmed by letter or fax. The sending of e-mails to
us does not have any period-protecting effect. Thank you for your
cooperation.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
