ERROR: All active_shards are unassigned_shards in ELK 5.4

Elastic Stack Elasticsearch
1

Hi Team,

I have built ELK setup 5.4 in my dev Environment.

I am using ELK 5.4 version and beats 5.4 version.

i can see the shards which are allocated i mean all active shards are of unassigned shards and it shows status as yellow. please find below details when i check cluster health.

{
"cluster_name" : "AON_EMEA_ECM_ELK",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 110,
"active_shards" : 110,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 110,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 50.0
}

Also we can see there comes error with below message regarding shards too when i load dashboard of metricbeat.

Courier fetch: 10 of 85 shards failed.

please find the below error message and let us know what would be the issue for this.

Please help us in making all active shards as assigned.

2

Hello,

Check Elasticsearch logs for any errors.

You can also use Cluster Allocation Explain API to find out why shards are unassigned.

Cheers

3

Was you only have one node in the cluster, Elasticsearch can only allocate the primary shards (which are all allocated) and will leave all configured replica shards unassigned. This is because Elasticsearch will not place two copies of the same shard on the same node. This is generally fine and not necessarily the cause behind the fetch error you are seeing. As the replica shards will never be allocated, you can change the index settings to not have a replica. Another, and better way to resolve the problem, would be to add more nodes to the cluster which would allow for replication and improve resiliency.

What is the specification of the host you are running this node on?

5

Hello Thiago,

As per cluster allocation explain API, im getting all shards related info since im using only 1 node for now, it says as for each indices im getting below shards and replicas which i need to make all active shards to assigned or i need to add node to my cluster or i should not make replica shards to be allocated.

please do help.

"indices": {
    "heartbeat-2017.05.11": {
      "status": "yellow",
      "number_of_shards": 5,
      "number_of_replicas": 1,
      "active_primary_shards": 5,
      "active_shards": 5,
      "relocating_shards": 0,
      "initializing_shards": 0,
      "unassigned_shards": 5,
      "shards": {
        "0": {
          "status": "yellow",
          "primary_active": true,
          "active_shards": 1,
          "relocating_shards": 0,
          "initializing_shards": 0,
          "unassigned_shards": 1
        },
        "1": {
          "status": "yellow",
          "primary_active": true,
          "active_shards": 1,
          "relocating_shards": 0,
          "initializing_shards": 0,
          "unassigned_shards": 1
        },
        "2": {
          "status": "yellow",
          "primary_active": true,
          "active_shards": 1,
          "relocating_shards": 0,
          "initializing_shards": 0,
          "unassigned_shards": 1
        },
        "3": {
          "status": "yellow",
          "primary_active": true,
          "active_shards": 1,
          "relocating_shards": 0,
          "initializing_shards": 0,
          "unassigned_shards": 1
        },
        "4": {
          "status": "yellow",
          "primary_active": true,
          "active_shards": 1,
          "relocating_shards": 0,
          "initializing_shards": 0,
          "unassigned_shards": 1
        }
6

thanks Christian, Yes im using only 1 node currently may be that cause an replica and making unassigned shards.

Let me know how do i change the index settings to not have a replica.

Since then i'm using ELK 5.4 in windows 2012 server host

Please do help me how do i add more nodes into my cluster. that would be more useful for us.

7

Hi Team,

Can anyone answer the solution for the above question please.

it will be much helpful.

Thank you

8

For the fetch error we would need to see some Elasticsearch logs. You could use pastebin for sending it.

9

please find the below elasticsearch logs.

[2017-06-01T11:30:10,116][ERROR][o.e.x.m.c.i.IndexRecoveryCollector] [bngwidap107.aonnet.aon.net] collector [index-recovery] failed to collect data
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:165) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.action.admin.indices.recovery.TransportRecoveryAction.checkGlobalBlock(TransportRecoveryAction.java:114) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.action.admin.indices.recovery.TransportRecoveryAction.checkGlobalBlock(TransportRecoveryAction.java:52) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.<init>(TransportBroadcastByNodeAction.java:256) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:234) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:79) ~[elasticsearch-5.4.0.jar:5.4.0]
[2017-06-01T11:30:10,618][ERROR][o.e.x.m.c.c.ClusterStatsCollector] [bngwidap107.aonnet.aon.net] collector [cluster-stats] failed to collect data
org.elasticsearch.action.NoShardAvailableActionException: No shard available for [get [.ml-anomalies-filesystem_available_per_1min][data_counts][filesystem_available_per_1min-data-counts]: routing [null]]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:209) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.start(TransportSingleShardAction.java:186) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.doExecute(TransportSingleShardAction.java:95) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.doExecute(TransportSingleShardAction.java:59) ~[elasticsearch-5.4.0.jar:5.4.0]
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.