Red Health on Elasticsearch

Elastic Stack Elasticsearch
1

Hi,

Everyday i am facing red health on Elasticsearch. Then i am chencking the indices under index management and i am seeing one or two red metricbeat indices. After i marked them and i deleted them the cluster health comes to green again.
The annoying thing is that i am facing it everyday.

I am sendig you the GET /_cluster/allocation/explain?pretty in order to help me if there is another way to solve my problem from not deleting the red indices everyday.

{
  "index" : "metricbeat-7.2.0-2019.11.07",
  "shard" : 0,
  "primary" : true,
  "current_state" : "unassigned",
  "unassigned_info" : {
    "reason" : "ALLOCATION_FAILED",
    "at" : "2019-11-07T02:17:36.362Z",
    "failed_allocation_attempts" : 5,
    "details" : "failed shard on node [EmyFlH3zS3KkvJptV02pMQ]: failed to create shard, failure IOException[failed to obtain in-memory shard lock]; nested: ShardLockObtainFailedException[[metricbeat-7.2.0-2019.11.07][0]: obtaining shard lock timed out after 5000ms, previous lock details: [shard creation] trying to lock for [shard creation]]; ",
    "last_allocation_status" : "no"
  },
  "can_allocate" : "no",
  "allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes that hold an in-sync shard copy",
  "node_allocation_decisions" : [
    {
      "node_id" : "46jaKQPSRC2drwy-bAfhxw",
      "node_name" : "xh-gr-elastic-2",
      "transport_address" : "10.159.166.9:9300",
      "node_attributes" : {
        "ml.machine_memory" : "269930577920",
        "ml.max_open_jobs" : "20",
        "xpack.installed" : "true"
      },
      "node_decision" : "no",
      "store" : {
        "found" : false
      }
    },
    {
      "node_id" : "CBiU9s_BSai2oBfVeZjwiQ",
      "node_name" : "xh-fr-elastic-1",
      "transport_address" : "135.238.239.48:9300",
      "node_attributes" : {
        "ml.machine_memory" : "16654970880",
        "ml.max_open_jobs" : "20",
        "xpack.installed" : "true"
      },
      "node_decision" : "no",
      "store" : {
        "in_sync" : false,
        "allocation_id" : "m2xHAFGuTem6OjkRDKSAgw",
        "store_exception" : {
          "type" : "file_not_found_exception",
          "reason" : "no segments* file found in SimpleFSDirectory@/data/elasticsearch/nodes/0/indices/UnMH4LvlSzaJS5-re4LYvg/0/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@76de86b6: files: [write.lock]"
        }
      }
    },
    {
      "node_id" : "EmyFlH3zS3KkvJptV02pMQ",
      "node_name" : "xh-gr-elastic-1",
      "transport_address" : "10.158.67.175:9300",
      "node_attributes" : {
        "ml.machine_memory" : "16654884864",
        "ml.max_open_jobs" : "20",
        "xpack.installed" : "true"
      },
      "node_decision" : "no",
      "store" : {
        "in_sync" : true,
        "allocation_id" : "mPCOxZWzQZitRDbk8WAQ2w"
      },
      "deciders" : [
        {
          "decider" : "max_retry",
          "decision" : "NO",
          "explanation" : "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2019-11-07T02:17:36.362Z], failed_attempts[5], delayed=false, details[failed shard on node [EmyFlH3zS3KkvJptV02pMQ]: failed to create shard, failure IOException[failed to obtain in-memory shard lock]; nested: ShardLockObtainFailedException[[metricbeat-7.2.0-2019.11.07][0]: obtaining shard lock timed out after 5000ms, previous lock details: [shard creation] trying to lock for [shard creation]]; ], allocation_status[deciders_no]]]"
        }
      ]
    },
    {
      "node_id" : "bLV-av6fSAWBTep4FKAqWA",
      "node_name" : "xh-gr-elastic-3",
      "transport_address" : "10.158.67.107:9300",
      "node_attributes" : {
        "ml.machine_memory" : "17179332608",
        "ml.max_open_jobs" : "20",
        "xpack.installed" : "true"
      },
      "node_decision" : "no",
      "store" : {
        "found" : false
      }
    },
    {
      "node_id" : "g9M0MIndTBW62pSrEdktzA",
      "node_name" : "xh-it-elastic-1",
      "transport_address" : "151.98.17.60:9300",
      "node_attributes" : {
        "ml.machine_memory" : "34359738368",
        "ml.max_open_jobs" : "20",
        "xpack.installed" : "true"
      },
      "node_decision" : "no",
      "store" : {
        "found" : false
      }
    },
    {
      "node_id" : "obT4-IMdTU6kOAHKVFNxzA",
      "node_name" : "xh-fr-elastic-2",
      "transport_address" : "135.238.239.132:9300",
      "node_attributes" : {
        "ml.machine_memory" : "269930713088",
        "ml.max_open_jobs" : "20",
        "xpack.installed" : "true"
      },
      "node_decision" : "no",
      "store" : {
        "in_sync" : false,
        "allocation_id" : "vLLhXHzdSJuw83F3_urnQg"
      }
    }
  ]
}
2

8

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.