Index in red

[Yaroslav_Glodov]

Elasticsearch not working periodically.
Version: 5.5.0

Error:

curl http://localhost:9200/_cluster/allocation/explain?pretty
{
  "index" : "donprinton",
  "shard" : 0,
  "primary" : true,
  "current_state" : "unassigned",
  "unassigned_info" : {
    "reason" : "ALLOCATION_FAILED",
    "at" : "2018-05-04T09:14:46.875Z",
    "failed_allocation_attempts" : 5,
    "details" : "failed to create shard, failure IOException[No space left on device]",
    "last_allocation_status" : "no"
  },
  "can_allocate" : "no",
  "allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes that hold an in-sync shard copy",
  "node_allocation_decisions" : [
    {
      "node_id" : "Su43ATrCTRym7FMPqZACbQ",
      "node_name" : "Su43ATr",
      "transport_address" : "127.0.0.1:9300",
      "node_decision" : "no",
      "store" : {
        "in_sync" : true,
        "allocation_id" : "2rBn4EEnQSKE4_q0PF5IwA"
      },
      "deciders" : [
        {
          "decider" : "max_retry",
          "decision" : "NO",
          "explanation" : "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-04T09:14:46.875Z], failed_attempts[5], delayed=false, details[failed to create shard, failure IOException[No space left on device]], allocation_status[deciders_no]]]"
        }
      ]
    }
  ]
}

Disk usage information (200G available):

df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            3.9G     0  3.9G   0% /dev
tmpfs           788M   94M  694M  12% /run
/dev/sda2       909G  709G  201G  78% /
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sda1       922M  897M  9.6M  99% /boot
tmpfs           788M     0  788M   0% /run/user/1001

Index size (26M only):

du -sh /usr/local/etc/elasticsearch/data/
26M	/usr/local/etc/elasticsearch/data/

[atira (Attila Boncok)]

@Yaroslav_Glodov

allocation is not permitted

How do permissions look like? Does the user who runs Elasticsearch have write permissions for that folder?

[Yaroslav_Glodov]

Elastic search is running under www-data, all /data/ folder is under www-data, user which running is www-data.
It does work for days and occasionally just stop working.
I run re-index (deleting old index and creating new one) and it works for next few days. So, I used simple solution - by cron I re-index every night whole database. It helped for few weeks, but suddenly it started to stop working again.

[atira (Attila Boncok)]

What do the logs show?
I had allocation problems related to virtual memory.