Issue in elastic search after moving data dir

Elastic Stack Elasticsearch
1

Hi

I installed elastic search on linux vm. Soon I figured my space was full. SO i stopped Elastic search and moved contents of data dir to a different dir and made changes in elasticsearch.yml file and restarted it. But I am getting error on kibana and elastic logs.

Configuration I made;
# ----------------------------------- Paths ------------------------------------
# Path to directory where to store the data (separate multiple locations by comma):
path.data: /path1/Elastic/data,/path2/Elastic/data
#
# Path to log files:
#
path.logs: /path1/Elastic/logs



Kibana error:
{"message":"all shards failed: [search_phase_execution_exception] all shards failed","statusCode":503,"error":"Service Unavailable"}

Elastic Search error:

[2019-03-31T23:02:19,152][WARN ][o.e.x.m.e.l.LocalExporter] [jG4MSlo] unexpected error while indexing monitoring document
org.elasticsearch.xpack.monitoring.exporter.ExportException: UnavailableShardsException[[.monitoring-kibana-6-2019.04.01][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.monitoring-kibana-6-2019.04.01][0]] containing [index {[.monitoring-kibana-6-2019.04.01][doc][7UR812kBRrsTOLH94OUr], source[{"cluster_uuid":"T3FxB08PRy-Le_3uRl3Zag","timestamp":"2019-04-01T06:01:19.091Z","interval_ms":10000,"type":"kibana_stats","source_node":{"uuid":"jG4MSlobSo2gNfA8Qdlzfw","host":"10.44.10.140","transport_address":"10.44.10.140:9300","ip":"10.44.10.140","name":"jG4MSlo","timestamp":"2019-04-01T06:01:19.096Z"},"kibana_stats":{"kibana":{"uuid":"7a5b28f3-6925-41df-923c-5c3c40aea7ab","name":"example.com","index":".kibana","host":"0.0.0.0","transport_address":"0.0.0.0:5601","version":"6.6.1","snapshot":false,"status":"red"},"usage":{"apm":{"has_any_services":false,"services_per_agent":{}},"infraops":{"last_24_hours":{"hits":{"infraops_hosts":13,"infraops_docker":0,"infraops_kubernetes":0,"logs":0}}},"xpack":{"reporting":{"available":true,"enabled":true,"browser_type":"chromium","_all":0,"csv":{"available":true,"total":0},"PNG":{"available":false,"total":0},"printable_pdf":{"available":false,"total":0},"status":{},"lastDay":{"_all":0,"csv":{"available":true,"total":0},"PNG":{"available":false,"total":0},"printable_pdf":{"available":false,"total":0},"status":{}},"last7Days":{"_all":0,"csv":{"available":true,"total":0},"PNG":{"available":false,"total":0},"printable_pdf":{"available":false,"total":0},"status":{}}}}}}}]}]]]


 [netty-transport-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) [netty-common-4.1.32.Final.jar:4.1.32.Final]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
[2019-03-31T22:49:41,731][WARN ][r.suppressed             ] [jG4MSlo] path: /.kibana/_search, params: {size=10000, ignore_unavailable=true, index=.kibana, filter_path=hits.hits._source.canvas-workpad}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:293) ~[elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:133) ~[elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:254) ~[elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:101) ~[elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.action.search.InitialSearchPhase.lambda$performPhaseOnShard$1(InitialSearchPhase.java:209) ~[elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.action.search.InitialSearchPhase$1.doRun(InitialSearchPhase.java:188) [elasticsearch-6.6.1.jar:6.6.1]
2

What does GET _cluster/health report? I suspect it will indicate that you have unassigned shards. If so, use the allocation explain API to find out why.

3

Before running this. I have deleted some of the Elastic Search indices which were in red status.

Now after running this I see this response.

{
"cluster_name" : "elasticsearch",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 85,
"active_shards" : 85,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 69,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 55.1948051948052
}

4

The cluster is red because at least one index is red and you can look for those indices in a red state using GET /_cat/indices?v&health=red

There are quite a lot of unassigned shards of which most will be replicas and with a 1 node cluster those cannot be assigned. So even with all active primaries you will at best have a yellow cluster health.

5

@Janko
I see the below response when I run the code block you provided. I manually checked the incices status in Management elastic search indices and I see all in yellow color.

{
"error": {
"root_cause": [
{
"type": "illegal_argument_exception",
"reason": "request [/_cat/indices] contains unrecognized parameter: [amp]"
}
],
"type": "illegal_argument_exception",
"reason": "request [/_cat/indices] contains unrecognized parameter: [amp]"
},
"status": 400
}

I am currently trying to setup my elastic stack to monitor some 500 vms. Do you think its a bad idea to run elastic search on a single node.

6

We haven't seen the output of the allocation explain API yet:

There was a copy-paste error, it should have read GET /_cat/indices?v&health=red. But really we want to be looking at the allocation explain API here.

It depends. How bad would it be if that single node were to fail and lose your data?

Generally you want multiple nodes for resilience; also if a single node can't handle the load then you can scale out by adding more nodes. But it's not required, and isn't obviously the cause of the problem you're asking about here.

7

Here is the output for allocation API.

.store.size
red open .monitoring-kibana-6-2019.03.31 _vxiwDgPT2WgZAOld-NfMQ 1 0
red open .monitoring-es-6-2019.03.31 yubxdTvTTo2PdwrrI_j06A 1 0

8

The allocation explain API is GET /_cluster/allocation/explain. The output you've shared is from GET _cat/allocation which isn't much help here.

9

Result for GET /_cluster/allocation/explain

{
"index" : "metricbeat-6.7.0-2019.04.01",
"shard" : 0,
"primary" : false,
"current_state" : "unassigned",
"unassigned_info" : {
"reason" : "CLUSTER_RECOVERED",
"at" : "2019-04-02T02:52:18.410Z",
"last_allocation_status" : "no_attempt"
},
"can_allocate" : "no",
"allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes",
"node_allocation_decisions" : [
{
"node_id" : "jG4MSlobSo2gNfA8Qdlzfw",
"node_name" : "jG4MSlo",
"transport_address" : "10.44.10.140:9300",
"node_attributes" : {
"ml.machine_memory" : "7674531840",
"xpack.installed" : "true",
"ml.max_open_jobs" : "20",
"ml.enabled" : "true"
},
"node_decision" : "no",
"deciders" : [
{
"decider" : "same_shard",
"decision" : "NO",
"explanation" : "the shard cannot be allocated to the same node on which a copy of the shard already exists [[metricbeat-6.7.0-2019.04.01][0], node[jG4MSlobSo2gNfA8Qdlzfw], [P], s[STARTED], a[id=pSuq11eyQf-NV5_eapKWwA]]"
}
]
}
]
}

I manually deleted few kibana indexes which I was not using. Do you think that might have caused this issue?

10

No, this particular shard is unassigned because it's a replica; the primary is allocated, but you only have one node so there's nowhere to put a replica. You should set number_of_replicas: 0 on this index.

We're interested in the unassigned primary shards, because this is the cause of your red indices, apparently monitoring-kibana-6-2019.03.31 and monitoring-es-6-2019.03.31. I don't know which primary shard(s) are unassigned within these indices, but you can find this out with GET _cat/shards and then ask the allocation explain API about those specific shards:

GET /_cluster/allocation/explain
{
  "index": "myindex",
  "shard": 0,
  "primary": true
}
11

When I ran the command GET_cat/shards

Got many results as many unassigned status. This is one of the results.

metricbeat-6.7.0-2019.04.01     0 r UNASSIGNED 

So used your query 

GET /_cluster/allocation/explain
{
  "index": "metricbeat-6.7.0-2019.04.01",
  "shard": 0,
  "primary": true
}

But i get the output as this. Is this correct.

{
  "index" : "metricbeat-6.7.0-2019.04.01",
  "shard" : 0,
  "primary" : true,
  "current_state" : "started",
  "current_node" : {
    "id" : "jG4MSlobSo2gNfA8Qdlzfw",
    "name" : "jG4MSlo",
    "transport_address" : "10.44.10.140:9300",
    "attributes" : {
      "ml.machine_memory" : "7674531840",
      "xpack.installed" : "true",
      "ml.max_open_jobs" : "20",
      "ml.enabled" : "true"
    },
    "weight_ranking" : 1
  },
  "can_remain_on_current_node" : "yes",
  "can_rebalance_cluster" : "no",
  "can_rebalance_cluster_decisions" : [
    {
      "decider" : "rebalance_only_when_active",
      "decision" : "NO",
      "explanation" : "rebalancing is not allowed until all replicas in the cluster are active"
    },
    {
      "decider" : "cluster_rebalance",
      "decision" : "NO",
      "explanation" : "the cluster has unassigned shards and cluster setting [cluster.routing.allocation.allow_rebalance] is set to [indices_all_active]"
    }
  ],
  "can_rebalance_to_other_node" : "no",
  "rebalance_explanation" : "rebalancing is not allowed"
}
12

No, that primary is also assigned.

In this line here the r indicates it's a replica that's unassigned; you only have one node so there's nowhere to put any replicas. You should set number_of_replicas: 0 on this index (or add a node)

metricbeat-6.7.0-2019.04.01     0 r UNASSIGNED

You are looking for an unassigned primary, which will have a p instead of the r.

1 Like
13

Is there a setting in elastic search or beats configuration where I can set this up. Instead of doing this manually on each of the index.

Or I can do it manually this one time but I might be installing more beats. Is there a way I can make these settings in advance?

14

I am unable to edit this settings any idea where I can make this change.

image
image.png869×498 26.2 KB

15

Do you mean the number_of_replicas setting? In Elasticsearch this defaults to 1 but can be changed with an index template. Afraid I'm the wrong person to ask about adjusting this through the UI.

You can set this on all existing indices with

PUT /_all/_settings
{"index.number_of_replicas":0}

However if you want to remove replicas from all future indices too then you will need to set this in a template somewhere.

16

Thank you very much for help. I am new to Elastic stack hopefully will gain some knowledge soon.

this worked and I can see all the indices as assigned.
PUT /_all/_settings
{"index.number_of_replicas":0}

However I see some of them unassigned. I have not created these indices. Does ES and Kibana create these.
.monitoring-kibana-6-2019.03.31 0 p UNASSIGNED
.monitoring-kibana-6-2019.03.31 0 p UNASSIGNED
.monitoring-es-6-2019.03.31 0 p UNASSIGNED

17

They look like indices created by the monitoring feature. I will admit that I'm not too familiar with monitoring, but I do not think these indices are created by default; you have to explicitly enable monitoring to see these indices being created. I guess the -es- index will be because xpack.monitoring.collection.enabled: true, and the -kibana- one will be because Kibana monitoring is also enabled.

18

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.