Cluster falling into YELLOW state during use of Snapshot API

Elastic Stack Elasticsearch
1

Our Elasticsearch cluster is comprised of 4 very large instances running
Elasticsearch 1.4.0.
The Snapshot API is set up to take backups once a day in the evening. We
maintain 5 snapshots at a time, expiring the old ones as they drop off.

Note that we have a few thousand indices in the cluster.

While the Snapshot API is in progress, many times the cluster will fall
into a yellow state. Here's an example of the output from GET
_cluster/health?pretty=true when this happens:

{
"cluster_name": "elasticsearch-cluster",
"status": "yellow",
"timed_out": false,
"number_of_nodes": 4,
"number_of_data_nodes": 4,
...
"relocating_shards": 0,
"initializing_shards": 1,
"unassigned_shards": 1
}

It would appear the Snapshot API is somehow interfering with shards
initialization and/or allocation while the Snapshot is in progress. Once
the cluster falls into the YELLOW state during the Snapshot, it stays in
the YELLOW state until the snapshot completes. When the Snapshot completes,
the cluster immediately switches back to a GREEN state after all unassigned
shards are allocated to nodes.

Has anyone else experienced this behavior with Elasticsearch Snapshots? If
so, are there thoughts on what might be causing it? Was there a resolution?

Thanks,
Steven

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/90740d3e-67b5-4094-ae82-a2085425c5bf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

What do your ES logs show?

On 29 April 2015 at 10:02, Steven B sbancroft@expectlabs.com wrote:

Our Elasticsearch cluster is comprised of 4 very large instances running
Elasticsearch 1.4.0.
The Snapshot API is set up to take backups once a day in the evening. We
maintain 5 snapshots at a time, expiring the old ones as they drop off.

Note that we have a few thousand indices in the cluster.

While the Snapshot API is in progress, many times the cluster will fall
into a yellow state. Here's an example of the output from GET
_cluster/health?pretty=true when this happens:

{
"cluster_name": "elasticsearch-cluster",
"status": "yellow",
"timed_out": false,
"number_of_nodes": 4,
"number_of_data_nodes": 4,
...
"relocating_shards": 0,
"initializing_shards": 1,
"unassigned_shards": 1
}

It would appear the Snapshot API is somehow interfering with shards
initialization and/or allocation while the Snapshot is in progress. Once
the cluster falls into the YELLOW state during the Snapshot, it stays in
the YELLOW state until the snapshot completes. When the Snapshot completes,
the cluster immediately switches back to a GREEN state after all unassigned
shards are allocated to nodes.

Has anyone else experienced this behavior with Elasticsearch Snapshots?
If so, are there thoughts on what might be causing it? Was there a
resolution?

Thanks,
Steven

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/90740d3e-67b5-4094-ae82-a2085425c5bf%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/90740d3e-67b5-4094-ae82-a2085425c5bf%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-eejOCU6XESZR7%2BJhsavhaCn%3DsjsdSyQLvpRm2%2BaE3VQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.