# Issue in elastic search after moving data dir
**URL:** https://discuss.elastic.co/t/issue-in-elastic-search-after-moving-data-dir/174712
**Category:** Elasticsearch
**Created:** [April 1, 2019, 6:12am UTC](https://discuss.elastic.co/t/issue-in-elastic-search-after-moving-data-dir/174712 "2019-04-01T06:12:23Z")
**Posts on this page:** 18
**Page:** 1
### Author:  [@syedsfayaz](https://discuss.elastic.co/u/syedsfayaz)
#### Post date: [April 1, 2019, 6:12am UTC](https://discuss.elastic.co/t/issue-in-elastic-search-after-moving-data-dir/174712/1 "2019-04-01T06:12:23Z")
Hi
I installed elastic search on linux vm. Soon I figured my space was full. SO i stopped Elastic search and moved contents of data dir to a different dir and made changes in elasticsearch.yml file and restarted it. But I am getting error on kibana and elastic logs.
```
Configuration I made;
# ----------------------------------- Paths ------------------------------------
# Path to directory where to store the data (separate multiple locations by comma):
path.data: /path1/Elastic/data,/path2/Elastic/data
#
# Path to log files:
#
path.logs: /path1/Elastic/logs
Kibana error:
{"message":"all shards failed: [search_phase_execution_exception] all shards failed","statusCode":503,"error":"Service Unavailable"}
```
Elastic Search error:
```
[2019-03-31T23:02:19,152][WARN][o.e.x.m.e.l.LocalExporter] [jG4MSlo] unexpected error while indexing monitoring document
org.elasticsearch.xpack.monitoring.exporter.ExportException: UnavailableShardsException[[.monitoring-kibana-6-2019.04.01][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.monitoring-kibana-6-2019.04.01][0]] containing [index {[.monitoring-kibana-6-2019.04.01][doc][7UR812kBRrsTOLH94OUr], source[{"cluster_uuid":"T3FxB08PRy-Le_3uRl3Zag","timestamp":"2019-04-01T06:01:19.091Z","interval_ms":10000,"type":"kibana_stats","source_node":{"uuid":"jG4MSlobSo2gNfA8Qdlzfw","host":"10.44.10.140","transport_address":"10.44.10.140:9300","ip":"10.44.10.140","name":"jG4MSlo","timestamp":"2019-04-01T06:01:19.096Z"},"kibana_stats":{"kibana":{"uuid":"7a5b28f3-6925-41df-923c-5c3c40aea7ab","name":"example.com","index":".kibana","host":"0.0.0.0","transport_address":"0.0.0.0:5601","version":"6.6.1","snapshot":false,"status":"red"},"usage":{"apm":{"has_any_services":false,"services_per_agent":{}},"infraops":{"last_24_hours":{"hits":{"infraops_hosts":13,"infraops_docker":0,"infraops_kubernetes":0,"logs":0}}},"xpack":{"reporting":{"available":true,"enabled":true,"browser_type":"chromium","_all":0,"csv":{"available":true,"total":0},"PNG":{"available":false,"total":0},"printable_pdf":{"available":false,"total":0},"status":{},"lastDay":{"_all":0,"csv":{"available":true,"total":0},"PNG":{"available":false,"total":0},"printable_pdf":{"available":false,"total":0},"status":{}},"last7Days":{"_all":0,"csv":{"available":true,"total":0},"PNG":{"available":false,"total":0},"printable_pdf":{"available":false,"total":0},"status":{}}}}}}}]}]]]
[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) [netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
[2019-03-31T22:49:41,731][WARN][r.suppressed] [jG4MSlo] path: /.kibana/_search, params: {size=10000, ignore_unavailable=true, index=.kibana, filter_path=hits.hits._source.canvas-workpad}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:293) ~[elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:133) ~[elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:254) ~[elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:101) ~[elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.action.search.InitialSearchPhase.lambda$performPhaseOnShard$1(InitialSearchPhase.java:209) ~[elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.action.search.InitialSearchPhase$1.doRun(InitialSearchPhase.java:188) [elasticsearch-6.6.1.jar:6.6.1]
```
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [April 2, 2019, 6:23am UTC](https://discuss.elastic.co/t/issue-in-elastic-search-after-moving-data-dir/174712/2 "2019-04-02T06:23:26Z")
What does `GET _cluster/health` report? I suspect it will indicate that you have unassigned shards. If so, use the [allocation explain API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-allocation-explain.html) to find out why.
---
### Author:  [@syedsfayaz](https://discuss.elastic.co/u/syedsfayaz)
#### Post date: [April 2, 2019, 3:51pm UTC](https://discuss.elastic.co/t/issue-in-elastic-search-after-moving-data-dir/174712/3 "2019-04-02T15:51:24Z")
Before running this. I have deleted some of the Elastic Search indices which were in red status.
Now after running this I see this response.
{
"cluster\_name" : "elasticsearch",
"status" : "red",
"timed\_out" : false,
"number\_of\_nodes" : 1,
"number\_of\_data\_nodes" : 1,
"active\_primary\_shards" : 85,
"active\_shards" : 85,
"relocating\_shards" : 0,
"initializing\_shards" : 0,
"unassigned\_shards" : 69,
"delayed\_unassigned\_shards" : 0,
"number\_of\_pending\_tasks" : 0,
"number\_of\_in\_flight\_fetch" : 0,
"task\_max\_waiting\_in\_queue\_millis" : 0,
"active\_shards\_percent\_as\_number" : 55.1948051948052
}
---
### Author:  [@Janko](https://discuss.elastic.co/u/Janko)
#### Post date: [April 3, 2019, 5:21pm UTC](https://discuss.elastic.co/t/issue-in-elastic-search-after-moving-data-dir/174712/4 "2019-04-03T17:21:28Z")
The cluster is red because at least one index is red and you can look for those indices in a `red` state using `GET /_cat/indices?v&health=red`
There are quite a lot of unassigned shards of which most will be replicas and with a 1 node cluster those cannot be assigned. So even with all active primaries you will at best have a `yellow` cluster health.
---
### Author:  [@syedsfayaz](https://discuss.elastic.co/u/syedsfayaz)
#### Post date: [April 3, 2019, 5:59pm UTC](https://discuss.elastic.co/t/issue-in-elastic-search-after-moving-data-dir/174712/5 "2019-04-03T17:59:52Z")
@Janko
I see the below response when I run the code block you provided. I manually checked the incices status in Management elastic search indices and I see all in yellow color.
{
"error": {
"root\_cause": [
{
"type": "illegal\_argument\_exception",
"reason": "request [/\_cat/indices] contains unrecognized parameter: [amp]"
}
],
"type": "illegal\_argument\_exception",
"reason": "request [/\_cat/indices] contains unrecognized parameter: [amp]"
},
"status": 400
}
I am currently trying to setup my elastic stack to monitor some 500 vms. Do you think its a bad idea to run elastic search on a single node.
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [April 3, 2019, 6:12pm UTC](https://discuss.elastic.co/t/issue-in-elastic-search-after-moving-data-dir/174712/6 "2019-04-03T18:12:36Z")
We haven't seen the output of the allocation explain API yet:
> [@DavidTurner](#):
>
> I suspect it will indicate that you have unassigned shards. If so, use the [allocation explain API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-allocation-explain.html) to find out why.
> [@syedsfayaz](#):
>
> "reason": "request [/\_cat/indices] contains unrecognized parameter: [amp]"
There was a copy-paste error, it should have read `GET /_cat/indices?v&health=red`. But really we want to be looking at the allocation explain API here.
> [@syedsfayaz](#):
>
> Do you think its a bad idea to run Elasticsearch on a single node.
It depends. How bad would it be if that single node were to fail and lose your data?
Generally you want multiple nodes for resilience; also if a single node can't handle the load then you can scale out by adding more nodes. But it's not required, and isn't obviously the cause of the problem you're asking about here.
---
### Author:  [@syedsfayaz](https://discuss.elastic.co/u/syedsfayaz)
#### Post date: [April 3, 2019, 6:15pm UTC](https://discuss.elastic.co/t/issue-in-elastic-search-after-moving-data-dir/174712/7 "2019-04-03T18:15:40Z")
Here is the output for allocation API.
.store.size
red open .monitoring-kibana-6-2019.03.31 \_vxiwDgPT2WgZAOld-NfMQ 1 0
red open .monitoring-es-6-2019.03.31 yubxdTvTTo2PdwrrI\_j06A 1 0
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [April 3, 2019, 6:17pm UTC](https://discuss.elastic.co/t/issue-in-elastic-search-after-moving-data-dir/174712/8 "2019-04-03T18:17:25Z")
The allocation **explain** API is `GET /_cluster/allocation/explain`. The output you've shared is from `GET _cat/allocation` which isn't much help here.
---
### Author:  [@syedsfayaz](https://discuss.elastic.co/u/syedsfayaz)
#### Post date: [April 3, 2019, 6:19pm UTC](https://discuss.elastic.co/t/issue-in-elastic-search-after-moving-data-dir/174712/9 "2019-04-03T18:19:53Z")
Result for GET /\_cluster/allocation/explain
{
"index" : "metricbeat-6.7.0-2019.04.01",
"shard" : 0,
"primary" : false,
"current\_state" : "unassigned",
"unassigned\_info" : {
"reason" : "CLUSTER\_RECOVERED",
"at" : "2019-04-02T02:52:18.410Z",
"last\_allocation\_status" : "no\_attempt"
},
"can\_allocate" : "no",
"allocate\_explanation" : "cannot allocate because allocation is not permitted to any of the nodes",
"node\_allocation\_decisions" : [
{
"node\_id" : "jG4MSlobSo2gNfA8Qdlzfw",
"node\_name" : "jG4MSlo",
"transport\_address" : "10.44.10.140:9300",
"node\_attributes" : {
"ml.machine\_memory" : "7674531840",
"xpack.installed" : "true",
"ml.max\_open\_jobs" : "20",
"ml.enabled" : "true"
},
"node\_decision" : "no",
"deciders" : [
{
"decider" : "same\_shard",
"decision" : "NO",
"explanation" : "the shard cannot be allocated to the same node on which a copy of the shard already exists [[metricbeat-6.7.0-2019.04.01][0], node[jG4MSlobSo2gNfA8Qdlzfw], [P], s[STARTED], a[id=pSuq11eyQf-NV5\_eapKWwA]]"
}
]
}
]
}
I manually deleted few kibana indexes which I was not using. Do you think that might have caused this issue?
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [April 3, 2019, 6:38pm UTC](https://discuss.elastic.co/t/issue-in-elastic-search-after-moving-data-dir/174712/10 "2019-04-03T18:38:47Z")
> [@syedsfayaz](#):
>
> I manually deleted few kibana indexes which I was not using. Do you think that might have caused this issue?
No, this particular shard is unassigned because it's a replica; the primary is allocated, but you only have one node so there's nowhere to put a replica. You should set `number_of_replicas: 0` on this index.
We're interested in the unassigned _primary_ shards, because this is the cause of your red indices, apparently `monitoring-kibana-6-2019.03.31` and `monitoring-es-6-2019.03.31`. I don't know which primary shard(s) are unassigned within these indices, but you can find this out with `GET _cat/shards` and then ask the allocation explain API about those specific shards:
```auto
GET /_cluster/allocation/explain
{
"index": "myindex",
"shard": 0,
"primary": true
}
```
---
### Author:  [@syedsfayaz](https://discuss.elastic.co/u/syedsfayaz)
#### Post date: [April 3, 2019, 6:49pm UTC](https://discuss.elastic.co/t/issue-in-elastic-search-after-moving-data-dir/174712/11 "2019-04-03T18:49:40Z")
When I ran the command GET\_cat/shards
Got many results as many unassigned status. This is one of the results.
```
metricbeat-6.7.0-2019.04.01 0 r UNASSIGNED
So used your query
GET /_cluster/allocation/explain
{
"index": "metricbeat-6.7.0-2019.04.01",
"shard": 0,
"primary": true
}
```
But i get the output as this. Is this correct.
```
{
"index" : "metricbeat-6.7.0-2019.04.01",
"shard" : 0,
"primary" : true,
"current_state" : "started",
"current_node" : {
"id" : "jG4MSlobSo2gNfA8Qdlzfw",
"name" : "jG4MSlo",
"transport_address" : "10.44.10.140:9300",
"attributes" : {
"ml.machine_memory" : "7674531840",
"xpack.installed" : "true",
"ml.max_open_jobs" : "20",
"ml.enabled" : "true"
},
"weight_ranking" : 1
},
"can_remain_on_current_node" : "yes",
"can_rebalance_cluster" : "no",
"can_rebalance_cluster_decisions" : [
{
"decider" : "rebalance_only_when_active",
"decision" : "NO",
"explanation" : "rebalancing is not allowed until all replicas in the cluster are active"
},
{
"decider" : "cluster_rebalance",
"decision" : "NO",
"explanation" : "the cluster has unassigned shards and cluster setting [cluster.routing.allocation.allow_rebalance] is set to [indices_all_active]"
}
],
"can_rebalance_to_other_node" : "no",
"rebalance_explanation" : "rebalancing is not allowed"
}
```
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [April 3, 2019, 6:57pm UTC](https://discuss.elastic.co/t/issue-in-elastic-search-after-moving-data-dir/174712/12 "2019-04-03T18:57:41Z")
No, that primary is also assigned.
In this line here the `r` indicates it's a replica that's unassigned; you only have one node so there's nowhere to put any replicas. You should set `number_of_replicas: 0` on this index (or add a node)
```auto
metricbeat-6.7.0-2019.04.01 0 r UNASSIGNED
```
You are looking for an unassigned primary, which will have a `p` instead of the `r`.
---
### Author:  [@syedsfayaz](https://discuss.elastic.co/u/syedsfayaz)
#### Post date: [April 3, 2019, 7:00pm UTC](https://discuss.elastic.co/t/issue-in-elastic-search-after-moving-data-dir/174712/13 "2019-04-03T19:00:37Z")
Is there a setting in elastic search or beats configuration where I can set this up. Instead of doing this manually on each of the index.
Or I can do it manually this one time but I might be installing more beats. Is there a way I can make these settings in advance?
---
### Author:  [@syedsfayaz](https://discuss.elastic.co/u/syedsfayaz)
#### Post date: [April 3, 2019, 7:02pm UTC](https://discuss.elastic.co/t/issue-in-elastic-search-after-moving-data-dir/174712/14 "2019-04-03T19:02:37Z")
I am unable to edit this settings any idea where I can make this change.
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [April 3, 2019, 7:09pm UTC](https://discuss.elastic.co/t/issue-in-elastic-search-after-moving-data-dir/174712/15 "2019-04-03T19:09:47Z")
> [@syedsfayaz](#):
>
> Is there a setting in Elasticsearch or beats configuration where I can set this up. Instead of doing this manually on each of the index.
Do you mean the `number_of_replicas` setting? In Elasticsearch this defaults to `1` but can be changed with an [index template](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html). Afraid I'm the wrong person to ask about adjusting this through the UI.
You can set this on all existing indices with
```plaintext
PUT /_all/_settings
{"index.number_of_replicas":0}
```
However if you want to remove replicas from all future indices too then you will need to set this in a template somewhere.
---
### Author:  [@syedsfayaz](https://discuss.elastic.co/u/syedsfayaz)
#### Post date: [April 3, 2019, 7:15pm UTC](https://discuss.elastic.co/t/issue-in-elastic-search-after-moving-data-dir/174712/16 "2019-04-03T19:15:20Z")
Thank you very much for help. I am new to Elastic stack hopefully will gain some knowledge soon.
this worked and I can see all the indices as assigned.
PUT /\_all/\_settings
{"index.number\_of\_replicas":0}
However I see some of them unassigned. I have not created these indices. Does ES and Kibana create these.
.monitoring-kibana-6-2019.03.31 0 p UNASSIGNED
.monitoring-kibana-6-2019.03.31 0 p UNASSIGNED
.monitoring-es-6-2019.03.31 0 p UNASSIGNED
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [April 3, 2019, 7:22pm UTC](https://discuss.elastic.co/t/issue-in-elastic-search-after-moving-data-dir/174712/17 "2019-04-03T19:22:28Z")
They look like indices created by the [monitoring feature](https://www.elastic.co/guide/en/elastic-stack-overview/6.7/xpack-monitoring.html). I will admit that I'm not too familiar with monitoring, but I do not think these indices are created by default; you have to explicitly enable monitoring to see these indices being created. I guess the `-es-` index will be because `xpack.monitoring.collection.enabled: true`, and the `-kibana-` one will be because Kibana monitoring is also enabled.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [May 1, 2019, 7:32pm UTC](https://discuss.elastic.co/t/issue-in-elastic-search-after-moving-data-dir/174712/18 "2019-05-01T19:32:54Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.