Index share allocation question with Docker containers

Elastic Stack Elasticsearch
1

Hi all,

I am currently working on building out an Elasticsearch cluster. I have sent out for the hardware that I'm planning to use and created another topic with the architecture that I'm planning and hardware specs to see if there were any suggestions. One thing that came up was the use of Docker to run multiple Elasticsearch instances on my servers because I have 128Gb of RAM on each (cuz ram is cheap) and it is not suggested that you go over a 32GGB heap size. I have figured out how to make this all work in a virtual environment but my current concern is shard allocation.

I currently have planed to use 2 shards with 1 replica. I planed for this because I have 4 hot servers and with this allocation I would lose 2 servers worth of storage space in effect due to the replica shards but I can also completely lose 2 hosts without any actual data loss. This is important because the hot nodes are all NVME storage so no RAID. They are SSD's so more reliable but a single disk failure is going to completely destroy any data on that host. My concern is that if I set up each data node to run say 3 docker containers with a 32GB heap each that some of the indexes may end up storing a primary shard and its replica on 2 containers on the same host. A failure of that host in that case would cause unrecoverable data loss for those indexes.

My question is if there is a way to prevent that from happening at all? Or is ES just smart enough not to do something like that by understanding that it is in a container on host1 so do not store replicas on containers in that same host? I would rather not start increasing the number of replicas because that is going to further decrease the amount of logs that I can store on the hot storage hosts and since we have never tried centralizing our logs, we are just making a wild guess at how much storage we need. I may make some edits to the number of shards or replicas later once we have a better idea of how much of this storage we are going to use up.

2

Hi @InnerJoin

You seem well on your way so you can do that with shard allocation see here in your case think of those underlying hosts like rack / rack awareness

3

Ohh also look close at the docs for JVM Heap it is not really 32GB is typically somewhere between 28-30GB

4

Great. Thanks for the information. And I'll down the memory allocation to somewhere around 29 or 30 GB if that is the suggestion there.

If I am understanding the documentation you linked correctly then I can set a custom attribute in elasticsearch.yml with:
node.attr.physicalhost: host01

Then tell all of the nodes that this attribute should be used for shard allocation awareness with:
cluster.routing.allocation.awareness.attributes: physicalhost

Am I understanding that procedure correctly or do I need to use the rack_id attribute that they are showing?

5

Yup that is pretty much it. Give it a try.

Back to JVM , my recommendation....don't guess follow the docs and figure out where the compressed object pointer are, if you want to guess start at 26

From the docs....

Set Xmx and Xms to no more than the threshold that the JVM uses for compressed object pointers (compressed oops). The exact threshold varies but is near 32 GB. You can verify that you are under the threshold by looking for a line in the logs like the following:

6

Your post got cut off there but I see where the docs say to look for the logs. I'm planning to log to a file instead of stdout so I should be able to grep that out when I get the hardware in and can test on that. Even setting it to 26GB per node is getting me a much larger heap than if I didn't use docker to split everything out so that's fine.

How can I verify where shards and replicas are being placed? I have a small 2 host virtual setup with 3 docker containers per node and kibana running. I can make the edits to the elasticsearch.yml files that I'm bind mounting but want to double-check that things are operating the way I expect with the shard allocation. Sorry but I'm pretty new with the platform and haven't set up or worked with a prod cluster yet.

7

I only pasted in part of the docs, I figured you could go to the docs and read the rest.

To see where the shards are, run

GET _cat/shards?v

in the dev tools it will show you which nodes the shards are on.

You should get to know the _cat API that provides a lot of information about the cluster indices

8

Awesome. Thanks for the info. I'll run some tests and check out the api's more. I've been relying on kibana for most things but I should get to know the api more than basic stuff I've been using.

9

You can run those APIs through Kibana Dev Tools but yes.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.