Can we run two ES Instances on the same server

Elastic Stack Elasticsearch
1

Currently I have one cluster 1 node ES .

I need few inputs on below points..

  • Can we run two ES instances on the same server ?
    if yes, how can we ensure, that the client have single endpoint .

  • Is there any reverse proxy used in ES and how we can ensure that back-end nodes are load balanced.

Please provide your inputs for the same.

1 Like
2

Why do you want to run more than one node on a single server? Is it a very large host?

3

We have numa enabled server with 2 sockets, each socket with 28 cores.

We have observed below points:

Under the same workload
Scenario 1
when we run ES7.3.0 on both the sockets (112 vcpus) , we can see CPU utilization is 30% , search throughput is 1700 ops/sec and when we increased the search load, the CPU utilization is not going beyond 30%. and search throughput is also not going beyond 1700 ops/sec.

Is there any Elasticsearch specific settings which is restricting to use all available CPU's(112 vcpus) on high search load?

Scenario 2:
When we bind ES7.3.0 to one socket (56 vcpus) we can see all the CPU's (56 vcpus) were utilized around 90 to 95%. (under same workload)
and the ops/sec is 2550 ops/sec which is more then 1700 ops/sec.

In case of 2 sockets (112 vcpus) why the CPU is not maxing out even with high search load.

As a next steps, we are planning to run 2 ES instance on both the sockets on same server.

Objective is to see, if we run 2 ES instance on the same server, will be above to see any gain in search throughput and CPU utilization under the same workload.

Can you please let me know whether we can run 2 ES instances on same machine ,
if yes, how the load balancing will happen and what kind of reverse proxy we are using in Elasticsearch.

4

Yes, but you will probably need to set it up manually.

You typically provide the client with a list of all nodes (with ports), so this is generally not necessary.

You can put a reverse proxy, e.g. nginx, in front of Elasticsearch, but this is generally not needed.

5

Thanks a lot for info ...

Can you please provide the steps for, how to run two ES on the same server?

6

Thanks a lot Christian for sharing your thoughts.

But I have a question.

When we create a cluster of ES nodes (spread across multiple servers), I believe we can get a single end point through reverse proxy, like nginx or apache http server. I saw your reply that though we can use reverse proxy, but this is generally not needed.

My question is, then how we can achieve a single end point or does ES provide any service similar? My thought on single end points is that it ensures load is balanced across multiple backend ES nodes and deployment is fault tolerant as well.

7

You should either create a service that runs via name-configuration
or you need to download the source(tar.gz or similar) and run the instance from the ~/bin/elasticsearch with a utility like tmux or screen. I mostly use the second option for running different elastic versions for demos. Just create a tmux session and run the instance you want, then detach and do that for the second one. Just make sure you are not saving data/logs on the same patch have 2 separate dirs of elastic with their own configs.
so

tmux new -s elastic1 > /opt/elastic1/bin/elasticsearch > :detach
tmux new -s elastic2 > /opt/elastic2/bin/elasticsearch >  :detach

As i said do not forget to edit their configs to save data and logs to different directories, ports and transports.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.