# Is there a way to place one or more elasticsearch shards on a separate disk?

**URL:** https://discuss.elastic.co/t/is-there-a-way-to-place-one-or-more-elasticsearch-shards-on-a-separate-disk/309797
**Category:** Elasticsearch
**Created:** [July 16, 2022, 4:07pm UTC](https://discuss.elastic.co/t/is-there-a-way-to-place-one-or-more-elasticsearch-shards-on-a-separate-disk/309797 "2022-07-16T16:07:36Z")
**Posts on this page:** 7
**Page:** 1

<div class="post-metadata">

### Author: ![jalustig](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jalustig/32/107161_2.png) [@jalustig](https://discuss.elastic.co/u/jalustig)
#### Post date: [July 16, 2022, 4:07pm UTC](https://discuss.elastic.co/t/is-there-a-way-to-place-one-or-more-elasticsearch-shards-on-a-separate-disk/309797/1 "2022-07-16T16:07:36Z")

</div>

I have a large elasticsearch index which is stored on an external blob storage device attached to my server, but I also have a large amount of free space on the main hard disk. Id like to put that space to use, if possible.

In order to more efficiently use resources, I thought that it might work to move one of the shards to the main hard disk. Is there a way to instruct ES, for instance, to put one shard on the main hard disk and two additional shards on the blob storage?

So far, I tried to do this manually by moving one shard’s directory to the main SSD and point the shard directory to the new location with a symlink, but ran into permissions issues. I stopped ES, and then copies the data with all permissions using `cp -R --preserve=all <index_directory>/0 <new_location_of_shard_directory_on_main_disk>`, and then pointed the `0` shard directory to the new location using a symlink: `ln -s 0 <new_location_of_shard_directory_on_main_disk>/0`.

This mostly worked, except that when I restarted elasticsearch, there were permissions issues: the log file issued an exception like this, with the key being that ES can't access the folder. I ensured that it was owned by `elasticsearch:elasticsearch` and that it had equal permissions as before.

```auto
Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/elastic-data/r6FYn6kPR1CuMwz49YQg_Q/0/_state/state-0.st" "read")
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:485) ~[?:?]
	at java.security.AccessController.checkPermission(AccessController.java:1068) ~[?:?]
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:411) ~[?:?]
	at java.lang.SecurityManager.checkRead(SecurityManager.java:751) ~[?:?]
	at sun.nio.fs.UnixChannelFactory.open(UnixChannelFactory.java:246) ~[?:?]
	at sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:133) ~[?:?]
	at sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:146) ~[?:?]
	at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:179) ~[?:?]
	at java.nio.channels.FileChannel.open(FileChannel.java:298) ~[?:?]
	at java.nio.channels.FileChannel.open(FileChannel.java:357) ~[?:?]
	at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:78) ~[lucene-core-9.1.0.jar:9.1.0 5b522487ba8e0f1002b50a136817ca037aec9686 - jtibs - 2022-03-16 10:32:40]
	at org.elasticsearch.gateway.MetadataStateFormat.read(MetadataStateFormat.java:303) ~[elasticsearch-8.2.2.jar:8.2.2]
	at org.elasticsearch.gateway.MetadataStateFormat.loadGeneration(MetadataStateFormat.java:428) ~[elasticsearch-8.2.2.jar:8.2.2]
	at org.elasticsearch.gateway.MetadataStateFormat.loadLatestStateWithGeneration(MetadataStateFormat.java:460) ~[elasticsearch-8.2.2.jar:8.2.2]
	at org.elasticsearch.gateway.MetadataStateFormat.loadLatestState(MetadataStateFormat.java:485) ~[elasticsearch-8.2.2.jar:8.2.2]
	at org.elasticsearch.gateway.TransportNodesListGatewayStartedShards.nodeOperation(TransportNodesListGatewayStartedShards.java:120) ~[elasticsearch-8.2.2.jar:8.2.2]
	at org.elasticsearch.gateway.TransportNodesListGatewayStartedShards.nodeOperation(TransportNodesListGatewayStartedShards.java:52) ~[elasticsearch-8.2.2.jar:8.2.2]
	at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:334) ~[elasticsearch-8.2.2.jar:8.2.2]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:67) ~[elasticsearch-8.2.2.jar:8.2.2]
	at org.elasticsearch.transport.TransportService$6.doRun(TransportService.java:918) ~[elasticsearch-8.2.2.jar:8.2.2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773) ~[elasticsearch-8.2.2.jar:8.2.2]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.2.2.jar:8.2.2]
	... 3 more

```

---

<div class="post-metadata">

### Author: ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [July 16, 2022, 5:26pm UTC](https://discuss.elastic.co/t/is-there-a-way-to-place-one-or-more-elasticsearch-shards-on-a-separate-disk/309797/2 "2022-07-16T17:26:03Z")

</div>

No, it is not possible, all the shards for a node needs to reside in the same `path.data` that is configured in `elasticsearch.yml`.

Also, you should not make any changes in the underlying files in the data directory, you risk corrupting your data and losing it if you do not have any snapshot.

---

<div class="post-metadata">

### Author: ![jalustig](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jalustig/32/107161_2.png) [@jalustig](https://discuss.elastic.co/u/jalustig)
#### Post date: [July 16, 2022, 5:51pm UTC](https://discuss.elastic.co/t/is-there-a-way-to-place-one-or-more-elasticsearch-shards-on-a-separate-disk/309797/3 "2022-07-16T17:51:57Z")

</div>

@leandrojmp I understand that it’s not recommended, but so I understand correctly, if all the data is there and the directory for the shard is just a symlink that points to another location, then it theoretically shouldn’t make a difference to Elasticsearch, right? ES would see `data-location/index-name/indexes/0` and it would just contain all the same data, but in another location.

---

<div class="post-metadata">

### Author: ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [July 16, 2022, 6:18pm UTC](https://discuss.elastic.co/t/is-there-a-way-to-place-one-or-more-elasticsearch-shards-on-a-separate-disk/309797/4 "2022-07-16T18:18:37Z")

</div>

In theory it should've work, Elasticsearch supports symlinks for the `path.data`, but I'm not sure how this would behave if you changed just one of the shards.

I don't think this is even tested because Elastic already strongly advise against manually chaging the data files.

If you already configured all the persmissions and it still doesn't work, you will need dig into the code to find the issue or wait to see if someone from Elastic can give some light on this because it is a very particular use case.

Also, in which version you are? There was some bug related to `path.data` and `symlinks` in versino 8 according to this [github issue](https://github.com/elastic/elasticsearch/issues/85701).

---

<div class="post-metadata">

### Author: ![jalustig](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jalustig/32/107161_2.png) [@jalustig](https://discuss.elastic.co/u/jalustig)
#### Post date: [July 16, 2022, 7:47pm UTC](https://discuss.elastic.co/t/is-there-a-way-to-place-one-or-more-elasticsearch-shards-on-a-separate-disk/309797/5 "2022-07-16T19:47:15Z")

</div>

@leandrojmp I’m currently on 8.2.

---

<div class="post-metadata">

### Author: ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [July 17, 2022, 8:21am UTC](https://discuss.elastic.co/t/is-there-a-way-to-place-one-or-more-elasticsearch-shards-on-a-separate-disk/309797/6 "2022-07-17T08:21:52Z")

</div>

Replacing a directory in the data path with a symlink is definitely not supported - see e.g. [these docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#data-path):

> **WARNING** : Don’t modify anything within the data directory or run processes that might interfere with its contents. If something other than Elasticsearch modifies the contents of the data directory, then Elasticsearch may fail, reporting corruption or other data inconsistencies, or may appear to work correctly having silently lost some of your data.

If you want to allocate different shards to different locations, run multiple nodes each with its own `path.data` and use [allocation filtering](https://www.elastic.co/guide/en/elasticsearch/reference/current/shard-allocation-filtering.html) to control which shards can be allocated to each node.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 14, 2022, 8:22am UTC](https://discuss.elastic.co/t/is-there-a-way-to-place-one-or-more-elasticsearch-shards-on-a-separate-disk/309797/7 "2022-08-14T08:22:14Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
