# Running out of space in /var/lib/elasticsearch
**URL:** https://discuss.elastic.co/t/running-out-of-space-in-var-lib-elasticsearch/148398
**Category:** Elasticsearch
**Created:** [September 13, 2018, 2:04am UTC](https://discuss.elastic.co/t/running-out-of-space-in-var-lib-elasticsearch/148398 "2018-09-13T02:04:57Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@popa](https://discuss.elastic.co/u/popa)
#### Post date: [September 13, 2018, 2:04am UTC](https://discuss.elastic.co/t/running-out-of-space-in-var-lib-elasticsearch/148398/1 "2018-09-13T02:04:57Z")
I currently have a single node Elastic Stack (with Winlogbeat) instance setup using mostly default settings.
We're running out of storage space, so we need to add another location for Elasticsearch data nodes to store indices.
I wanted to know what happens behind the scenes when I execute the following steps:
**Step 1:** Modify `/etc/elasticsearch/elasticsearch.yml` using your favorite editor and change:
```auto
path.data: /var/lib/elasticsearch
```
to something like:
```auto
path:
data:
- /var/lib/elasticsearch
- /mnt/elasticsearch_1
- /mnt/elasticsearch_2
- /mnt/elasticsearch_3
```
**Step 2:** Make sure `elasticsearch:elasticsearch` has access full access to `/mnt/elasticsearch_*` so that it can read, write, delete, etc.
**Step 3:** Restart the Elasticsearch service by running `systemctl restart elasticsearch`.
Here are my questions:
**Question 1:** What will happen to the data currently in `path.data: /var/lib/elasticsearch` ?
**Question 2:** Will the indices be distributed between the four location(s) or will they just be replicas?
**Question 3:** Is there a way to migrate the data from `/var/lib/elasticsearch` to the new location(s)?
I've come across a few threads, however, they're a bit old and wanted to make sure the steps were the same:
> [@How to move Data and Logs to new directory?](https://discuss.elastic.co/t/how-to-move-data-and-logs-to-new-directory/28382/3):
>
> Thanks Mark, but I was looking for more detailed steps. Here's what I ended up doing in case anyone else is looking for this information the future. \> • Determine current location of data files \> curl "localhost:9200/\_nodes/settings?pretty=true" • Create new directory structure - e.g; \> cd /spare \> mkdir data • Stop ntopng service \> systemctl stop ntopng.service • Stop ES service \> systemctl stop elasticsearch.service • Navigate to the current Data Directory determined in step 1 and cop…
> [@Running out of space on main disk - will altering path.data help?](https://discuss.elastic.co/t/running-out-of-space-on-main-disk-will-altering-path-data-help/61958):
>
> Hey all. I have a running single node ES that is running out of disk space. If I add another drive to the machine, format it and add it to the path.data so I have the line: path.data["/var/lib/elasticsearch/data", "/new/drive/mountpoint"] in the config file, will Elasticsearch start putting data on the new drive instead of the old drive thats running out of space? Or, would it be better to move the data in /var/lib/elasticsearch/data to the new larger drive and change the path.data to path.…
I'd test this out myself, however, I do not have a development box to work with.
Thanks for reading!
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [September 13, 2018, 2:07am UTC](https://discuss.elastic.co/t/running-out-of-space-in-var-lib-elasticsearch/148398/2 "2018-09-13T02:07:56Z")
1. Nothing, it will stay there
2. They will be distributed. You cannot have both primary and replicas shards existing on the same node
3. I have seen some people create a new location, manually `mv` things across and then update the config. That's not guaranteed to work, nor supported though.
---
### Author:  [@popa](https://discuss.elastic.co/u/popa)
#### Post date: [September 13, 2018, 2:18am UTC](https://discuss.elastic.co/t/running-out-of-space-in-var-lib-elasticsearch/148398/3 "2018-09-13T02:18:08Z")
Thanks for the quick reply, @warkolm.
Is there a supported way to move data from one location to another, or is this something that should have been considered at the beginning of the project?
Since this is only a proof of concept I'm working on, I can add it to my notes and plan my `elasticsearch.yml` config for production.
Thanks again for your help.
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [September 13, 2018, 2:32am UTC](https://discuss.elastic.co/t/running-out-of-space-in-var-lib-elasticsearch/148398/4 "2018-09-13T02:32:55Z")
> [@popa](#):
>
> Is there a supported way to move data from one location to another
The only way I can think is to add more nodes and then do a bunch of moving around. Which isn't ideal.
Can you remove some of the indices you don't need?
---
### Author:  [@popa](https://discuss.elastic.co/u/popa)
#### Post date: [September 13, 2018, 4:08am UTC](https://discuss.elastic.co/t/running-out-of-space-in-var-lib-elasticsearch/148398/5 "2018-09-13T04:08:44Z")
Sure can. I'll be using curator to manage my indices. However, the issue is that our Elastic Stack node was initially setup on a 30GB SSD. 😔
---
### Author:  [@popa](https://discuss.elastic.co/u/popa)
#### Post date: [September 19, 2018, 5:40am UTC](https://discuss.elastic.co/t/running-out-of-space-in-var-lib-elasticsearch/148398/6 "2018-09-19T05:40:03Z")
I was able to migrate my data without any apparent issues. Instead of copying, I used rsync, which is a safer alternative.
Here's what I did:
**Step 1:** Create a new directory on the new drive to store your indices:
```auto
mkdir /apps/elasticsearch
```
**Step 2:** Disable shard allocation.
When you shut down a node, the allocation process waits for index.unassigned.node\_left.delayed\_timeout (by default, one minute) before starting to replicate the shards on that node to other nodes in the cluster, which can involve a lot of I/O. Since the node is shortly going to be restarted, this I/O is unnecessary. You can avoid racing the clock by disabling allocation before shutting down the node:
```auto
PUT _cluster/settings
{
"persistent": {
"cluster.routing.allocation.enable": "none"
}
}
```
**Step 3:** Stop indexing and perform a synced flush.
Performing a synced-flush speeds up shard recovery.
You will likely need to stop Logstash or Winlogbeat from sending events to Elasticsearch to stop the failed status codes: `systemctl stop logstash`.
```auto
POST _flush/synced
```
When you perform a synced flush, check the response to make sure there are no failures. Synced flush operations that fail due to pending indexing operations are listed in the response body, although the request itself still returns a 200 OK status. If there are failures, reissue the request.
**Step 4:** Stop Elasticsearch:
```auto
systemctl stop elasticsearch
```
**Step 5:** Copy your indices to the new location:
```auto
rsync --info=progress2 -auvrz /var/lib/elasticsearch /apps/elasticsearch
```
or using copy (not recommended)
```auto
cd /var/lib/elasticsearch
cp -RP * /apps/elasticsearch
```
**Step 6:** Change the ownership on copied files and directorys to `elasticsearch:elasticsearch`:
```auto
chown -R elasticsearch:elasticsearch /apps/elasticsearch
```
This step is only necessary when using the copy command.
**Step 7:** Change the `path.data` location in `elasticsearch.yml` using your favorite editor:
```auto
path.data: /apps/elasticsearch
```
If you have multiple locations where you're storing indices, you can use the following syntax:
```auto
path:
data:
- /mnt/elasticsearch_1
- /mnt/elasticsearch_2
- /mnt/elasticsearch_3
```
**Step 9:** Start Elasticsearch:
```auto
systemctl start elasticsearch
```
**Step 10:** Reenable allocation.
When all nodes have joined the cluster and recovered their primary shards, reenable allocation by restoring cluster.routing.allocation.enable to its default:
```auto
PUT _cluster/settings
{
"persistent": {
"cluster.routing.allocation.enable": null
}
}
```
Once allocation is reenabled, the cluster starts allocating replica shards to the data nodes. At this point it is safe to resume indexing and searching, but your cluster will recover more quickly if you can wait until all primary and replica shards have been successfully allocated and the status of all nodes is green.
It will take several minutes (or longer) depending on how many documents you have stored in your indices.
More information on this can be found on the following documentation pages:
- [https://www.elastic.co/guide/en/elasticsearch/reference/current/restart-upgrade.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/restart-upgrade.html)
- [https://www.elastic.co/guide/en/elasticsearch/reference/current/path-settings.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/path-settings.html)
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 17, 2018, 5:40am UTC](https://discuss.elastic.co/t/running-out-of-space-in-var-lib-elasticsearch/148398/7 "2018-10-17T05:40:08Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.