Multiple instances of Elasitcsearch on single node

Elastic Stack Elasticsearch
1

Due to circumstances beyond my control, i need to start 2 instances on one physical server.
My servers specs are 32 cpu cores, 189GB ram, 16x1TB HDD and 2x120GB SSD.
I plan to have one instance run on the 16TB HDD partition and another instance on the 240GB SSD partition. Ultimate goal is to create a hot pool and cold pool.

The design is stuck given limitations so i am more curious about actual deployment / implementations. I dont see any clear cut guide about what to do. I am sure other people have tried this before. I have hoping to have a good "guide" or discussion here as a "peer review" and for others who need it in the future.

This is my thoughts on what needs to be done:

  • I install the primary instance using elasticsearch's yum repo
  • duplicate the init script changing LOG_DIR, DATA_DIR,WORK_DIR,CONF_DIR, CONF_FILE,PID_DIR
  • duplicate the elasticsearch.yml changing node.name, http.ports, path.data, transport.tcp.port, path.log, lower processors
  • create the proper data/log directories
  • (sysconfig file is mentioned but everything is commented out on my file so not sure if that file is still valid)
  • run the new init script and cross my fingers.
  • if the 2nd instance actually runs from what i can tell in the log, then add to chkconfig

upgrades would be super tricky i believe. I would need to stop both instances on the server. yum upgrade the binaries. Check to make sure none of the configs were messed up. Start the instances up one by one.

Would greatly appreciate anyone sharing their experiences. What worked / what didnt.

Thanks for reading this long post

2

No comment or discussion?

3

How many physical machines do you have? If you have 3 machines, then I'd run two separate Elasticsearch processes. One as a master node and one as a data node. You'll need to configure the JAVA_HEAP for each instance and set a minumim master node of 2. Given this kind of setup I'd use the tar.gz download and extract the installs into different directories. This will keep life simple and easier to manage both.

If you run 2 data nodes on one physical machine then you should set cluster.routing.allocation.same_shard.host to true to avoid allocating a primary and replica shard on the the same physical machine.

https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cluster.html#shards-allocation

4

8 physical servers dedicated as data nodes. Already have master nodes split off as VMs on separate servers.

Can you elaborate on this statement? How is it easier than yum install
"Given this kind of setup I'd use the tar.gz download and extract the installs into different directories. This will keep life simple and easier to manage both."

Thanks

5

Cause it's a bit harder to manage configs, services etc using a yum install.

If you leverage the Elasticsearch puppet module with a yum install it'll be easier though!