Unassigned shards status flip-flopping

Elastic Stack Elasticsearch
1

Hey there

I've noticed an odd thing. With ES-1.7.5 on CentOS7, "http://localhost:9200/_cluster/health" can show that there are no UNASSIGNED shards, but if I restart it, suddenly thousands show up

We're using graylog, and I've noticed it hanging - being unable to push new data into ES, and was confused why ES was showing "GREEN". And yet if I shut down graylog (ie stop new data), a restart of ES immediately goes to "RED" with thousands of unassigned shards - which certainly explains why graylog was struggling

ie

  1. curl http://localhost:9200/_cluster/health shows GREEN, no unassigned shards
  2. restart elasticsearch
  3. curl http://localhost:9200/_cluster/health shows RED, thousands of unassigned shards

If I then wait for ES to process those shards (which it does), then start graylog-server, everything works splendidly for some hours - until it happens again

I know next to nothing about ES, but that doesn't sounds right to me? Surely restarting it shouldn't cause counters to vary by that much?

This is a test server (ie graylog and ES on the same box), with 36G RAM and 24 cores, with no sign of load problems (and ES_HEAP_SIZE set to 17G), so I'm confused as to why anything is acting up. Hopefully this explanation prompts someone into telling me what stoopid noobie mistake I've made :slight_smile:

Thanks!

Jason