Unable to disable shard allocation when an index is stored in memory

Elastic Stack Elasticsearch
1

Hello,
I have been writing a script that for properly restarting all the nodes of
an ES cluster and I have noticed a few particular things in the process.

The process is quite simple, it is the same described in the official ES
documentation
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-nodes-shutdown.html#_rolling_restart_of_nodes_full_cluster_restart
For each node in the cluster:

  • disable shard allocation
  • shut down the node
  • restart the service via SSH
  • enable back shard allocation once the node rejoins the cluster
  • wait for the cluster to be OK before proceeding to the next node

However, one of the indices of the cluster is stored in memory.
When trying to disable shard allocation, we get an IllegalStateException
("Free bytes [X] cannot be less than 0 or greater than total bytes [Y]").

Such behaviour does not happens in a cluster when all the indices are
stored on disk.

When using 3 replicas for every shard, shutting down a node with an index
in memory should not be a problem, since even if the content is lost during
the shutdown, it should be able to retrieve the information from the
replicas when the node rejoins the cluster.

I do believe this is an issue from elasticsearch.
Can anyone confirm it or show me some light on the issue before opening an
issue?

Thanks,
Bruno

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/438ed2ff-b761-42c5-8d02-0855764e61a4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Can you elaborate on how you are storing this index in memory?

On 22 January 2015 at 08:17, bplsilva@chaordicsystems.com wrote:

Hello,
I have been writing a script that for properly restarting all the nodes of
an ES cluster and I have noticed a few particular things in the process.

The process is quite simple, it is the same described in the official ES
documentation

Elasticsearch Platform — Find real-time answers at scale | Elastic
For each node in the cluster:

  • disable shard allocation
  • shut down the node
  • restart the service via SSH
  • enable back shard allocation once the node rejoins the cluster
  • wait for the cluster to be OK before proceeding to the next node

However, one of the indices of the cluster is stored in memory.
When trying to disable shard allocation, we get an IllegalStateException
("Free bytes cannot be less than 0 or greater than total bytes [Y]").

Such behaviour does not happens in a cluster when all the indices are
stored on disk.

When using 3 replicas for every shard, shutting down a node with an index
in memory should not be a problem, since even if the content is lost during
the shutdown, it should be able to retrieve the information from the
replicas when the node rejoins the cluster.

I do believe this is an issue from elasticsearch.
Can anyone confirm it or show me some light on the issue before opening an
issue?

Thanks,
Bruno

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/438ed2ff-b761-42c5-8d02-0855764e61a4%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/438ed2ff-b761-42c5-8d02-0855764e61a4%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9C2QCogxununyt5cMjhk%2BqO_unLQsmcaEo2SX6800s-w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

As suggested here in the documentation
(Elasticsearch Platform — Find real-time answers at scale | Elastic),
we have performed the following command

curl -XPUT hostName:9200/indexName -d '{ "settings": { "index.store.type": "memory" } }';

when creating the index.

This increases our application performance, however our snapshooting
process takes longer, and, we haven't been able to disable shard allocation.

Bruno

Le mercredi 21 janvier 2015 18:51:41 UTC-2, Mark Walkom a écrit :

Can you elaborate on how you are storing this index in memory?

On 22 January 2015 at 08:17, <bpls...@chaordicsystems.com <javascript:>>
wrote:

Hello,
I have been writing a script that for properly restarting all the nodes
of an ES cluster and I have noticed a few particular things in the process.

The process is quite simple, it is the same described in the official ES
documentation

Elasticsearch Platform — Find real-time answers at scale | Elastic
For each node in the cluster:

  • disable shard allocation
  • shut down the node
  • restart the service via SSH
  • enable back shard allocation once the node rejoins the cluster
  • wait for the cluster to be OK before proceeding to the next node

However, one of the indices of the cluster is stored in memory.
When trying to disable shard allocation, we get an IllegalStateException
("Free bytes cannot be less than 0 or greater than total bytes [Y]").

Such behaviour does not happens in a cluster when all the indices are
stored on disk.

When using 3 replicas for every shard, shutting down a node with an index
in memory should not be a problem, since even if the content is lost during
the shutdown, it should be able to retrieve the information from the
replicas when the node rejoins the cluster.

I do believe this is an issue from elasticsearch.
Can anyone confirm it or show me some light on the issue before opening
an issue?

Thanks,
Bruno

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/438ed2ff-b761-42c5-8d02-0855764e61a4%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/438ed2ff-b761-42c5-8d02-0855764e61a4%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c6098728-0a9f-4b07-be7b-ab1a5bfbcfc5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.