13mil documents and Elasticsearch status=red after server restart

Elastic Stack Elasticsearch
1

Scenario:
Log files parsed by logstash generated 13mil results in elasticsearch.
After a server restart the shards are not allocated anymore.
The Roy Russo dashboard says there's a problem with the swap space.

I've tried setting mlockall to true, but then I found out that it only
works on Linux and not Windows?... And it had no effect.

I've found some conflicting information herehttp://gibrown.wordpress.com/2013/12/05/managing-elasticsearch-cluster-restart-time/and
herehttp://stackoverflow.com/questions/19967472/elasticsearch-unassigned-shards-how-to-fixregarding how to disable allocation (or at least it seams contradictory to
me...)

Any ideas? Thanks!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/11edbbd0-5305-4823-a854-114dfef410f4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

I've also closed all indices, then started one only and waited... Still
ended up with a red status and some unallocated shards.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/15a3da82-4730-415a-bf12-dd556d8b5177%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

3

How many nodes do you have? How many replicas?

After a quick glance, it looks like both stack overflow answers have the
correct way to disable allocation.

--
Ivan

On Wed, Apr 9, 2014 at 8:57 AM, Madeleine mxmadalina@gmail.com wrote:

I've also closed all indices, then started one only and waited... Still
ended up with a red status and some unallocated shards.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/15a3da82-4730-415a-bf12-dd556d8b5177%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/15a3da82-4730-415a-bf12-dd556d8b5177%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDm3a4%3DxqwvZXk%2B_hnkDU5vojjaf-RNsO%2BjpswbnnnTVw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

4

Hey,

can you check the log files for error messages (possibly also before the
restart)? Also the recovery API may help you do identify if you still
recovering shards or recovery is finished and you had problems while
recovering the data (again, checking the log files should help a lot)

--Alex

On Fri, Apr 11, 2014 at 1:27 AM, Ivan Brusic ivan@brusic.com wrote:

How many nodes do you have? How many replicas?

After a quick glance, it looks like both stack overflow answers have the
correct way to disable allocation.

--
Ivan

On Wed, Apr 9, 2014 at 8:57 AM, Madeleine mxmadalina@gmail.com wrote:

I've also closed all indices, then started one only and waited... Still
ended up with a red status and some unallocated shards.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/15a3da82-4730-415a-bf12-dd556d8b5177%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/15a3da82-4730-415a-bf12-dd556d8b5177%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDm3a4%3DxqwvZXk%2B_hnkDU5vojjaf-RNsO%2BjpswbnnnTVw%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDm3a4%3DxqwvZXk%2B_hnkDU5vojjaf-RNsO%2BjpswbnnnTVw%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGCwEM_AKG2ifzKEwH-aMcAVrDxSpnozAe-YKx9OCeQ9fG6WBA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

5

Sense says:
{
"count": 4640383,
"_shards": {
"total": 115,
"successful": 54,
"failed": 0
}
}

Roy Russo shows:

Nodes 1

Data Nodes 1

Active Primary Shards 54
Active Shards 54
Relocating Shards 0
Initializing Shards 0
Unassigned Shards 176

Problem is there is one node only shown by Roy Russo. There should be
more...

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/03c1babd-62bd-4796-b14a-95b0f4c811f1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

6

There is no error in the logs. Just information like this:
[2014-05-05 10:41:58,340][INFO ][gateway ] [Arides]
recovered [40] indices into cluster_state

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fe34f92a-c945-469e-879b-c57ac7e493b9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

7

You need to check the other nodes then.

Use a monitoring plugin as well, it's a lot easier.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 7 May 2014 20:47, Madeleine mxmadalina@gmail.com wrote:

There is no error in the logs. Just information like this:
[2014-05-05 10:41:58,340][INFO ][gateway ] [Arides]
recovered [40] indices into cluster_state

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/fe34f92a-c945-469e-879b-c57ac7e493b9%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/fe34f92a-c945-469e-879b-c57ac7e493b9%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624YpzhfLTMQ%2B7BovAzbjpVSrPmB6UzYm__7fE5CSz9oc2w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.