We currently have an elasticsearch cluster with two seperate equal servers as two data nodes - one master, one slave.
The servers each have 1 TB of diskspace. However, master is now at 81% of disk usage and slave is at 74%. I'm not sure why they are not equally used since each index and document should be present on both nodes but I read that this still can happen.
Problem:
We now decided to add an additional 1TB disk to each of the nodes before upgrading to a third server/node. In the /etc/elasticsearch/elasticsearch.yml files the additional path of the new disk was added.
Current configuration there:
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch,/elasticsearch_data
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
Furthermore the ownership of the new disk was edited via
sudo chmod -Rv 777 /elasticsearch_data
When shutting down elasticsearch nodes and restarting them all indices are green again, but the new disk stays for multiple weeks at just 75MB of usage while the usage of the original disks keeps growing.
When cd ing into the new created path /elasticsearch_data there is now a structure with /nodes/0/…. With indices and node.lock in there.
However, it continues to not add any data to that disk.
Am I missing anything and this behaviour is expected?
Thanks @warkolm you guys are doing an awesome job with elasticsearch!
No, currently there are no new indices created. But the way I understood the description it should also use new space for existing indices right? Otherwise the size of a single index would be limited by the size of the single disk it landed on initially. Is this the case?
When checking the second data path it shows in /elasticsearch_data/nodes/0/indices basically all nodes that I also get when calling http://myelasticserver/_cat/indices?. But, as said, size of diskspace in use is still only 75MB for weeks.
In the logs that I checked I was not able to find any Error in that regard. Also when calling http://95.217.108.253:9200/_cluster/stats?human&pretty I get for example
I think that's expected, Elasticsearch won't rebalance shards across the paths, and given that you only have two nodes it can't rebalance them across nodes either. Support for multiple data paths is being removed so it's unlikely this will ever be fixed now.
IMO a better solution is to add more nodes to use the extra disks. Alternatively you could use LVM or RAID or something to make a single filesystem that spans both disks, but that might be tricky if you aren't already using LVM.
Thanks for making me aware of that upcomming change. So what would be "correct" way of doing it then to be future proof? - Just adding more nodes seemed to be unnecessary high costs when I compare the costs of an elasticsearch node / server like we have it currently to the costs of a 1TB disk.
Would it be a working option to drop a bigger index and recreate it? Would it then get added to the new disk if there is more free space available?
By "node" I meant "Elasticsearch process" -- you can run more than one of these on the same physical host. However ...
... IMO the best way is to combine all your disks into a single volume using RAID or LVM.
Yes I would expect newly-created shards to be placed on the emptier disks. Maybe not all of them, there is some preference for putting different shards of the same index onto different disks, but certainly the first shard copy of each new index will go onto the emptier disk.
The physical location of each shard tends to be pretty well-hidden in the APIs. The only place I know to find it is in the output of the indices stats API if you set ?level=shards.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
