[ES2.2] how to run multiple elasticsearch nodes as processes on the same server

Elastic Stack Elasticsearch
1

hi guys,

i was wondering if you help me out here;
am trying to run multiple elasticsearch processes on the same (CentOS) server, but i have been un-successful so far.
and i have not enabled the service wrapper. and Elasticsearch has been installed using the .rpm package

the requirements are:

  1. every instance belongs to a different cluster (cluster.name)
  2. every instance uses a different port, 9201, 9202, 9203, etc.
  3. every instance should be parameterised with different ES_HEAP_SIZE

the elasticsearch.yml file is attached where all parameters are described.

and the questions are:

  1. how to set a different configuration file per instance when Des.config seems to be deprecated in 2.2
  2. how to set a custom ES_HEAP_SIZE (-Xmx=24G -Xms=24G) when
    # bin/elasticsearch -Des.config=config/IP-spotlight.RRv4/elasticsearch.yml 
    [2016-02-14 19:44:02,858][INFO ][bootstrap                ] es.config is no longer supported. elasticsearch.yml must be placed in the config directory and cannot be renamed.

please help ..

2

Hi nskalis!

I would use Docker to configure an environment like that. That approach has a lot of benefits.

The biggest one is that it would be pretty easy to set up and support.

Best,
Nick

3

thanks @molsznski i followed your approach. but i have an issue with the kibana (docker version);

could you please advise why; when using --net=host the port mapping does NOT work ?
for example,

"docker run -d --name=\"IP-spotlight.kibana.RRvpn\" --net=host --publish 5603:5601 -e ELASTICSEARCH_URL=http://nl-ams02c-ispdat02:9202 --label app=\"IP-spotlight\" --label component=\"kibana\" --label util=\"RRvpn\" cabana"

# telnet 127.0.0.1 5603
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused

# telnet 127.0.0.1 5601
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
^]
  • when running the kibana image without the --net=host port mapping works but cannot connect to the elastic search instance
  • i am running elasticsearch using the --net=host in order not for docker_proxy to struggle with high IO
4

Hi @nskalis!

Basically, you should not have any performance penalty for using the docker networking stack.
There are some great articles that stress test various docker networking strategies and, in a nutshell, you can have the goodies of both worlds if you need that at some point in time. I would not bother.

I am not a networking guy, though here are my thoughts based on some exp. When you use --net=host:

  • You are on your own with networking
  • I think, port mapping should be done via some additional script
  • You might want to check out official docker docker docs when it comes to -net=host, since this is not really a default docker way to do it

Here is a sample docker-compose I use for dev

`
elasticsearch:
build: elk/elasticsearch/
ports:
- "9200:9200"
- "9300:9300"
volumes:
- ./elk/elasticsearch/default_config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml

logstash:
image: logstash:2.2
ports:
- "5000:5000"
volumes:
- ./elk/logstash/config-dir:/config-dir
links:
- elasticsearch
command: "logstash -f config-dir/logstash.conf"

kibana:
image: kibana:4.4
ports:
- "5601:5601"
links:
- elasticsearch
`

5

I have not tried v2.2 so my suggestion might not work... if you do RPM installation, check the startup script at /etc/init.d/elasticsearch. That's where I set the config location such as CONF_DIR, CONF_FILE.

If this works, you can make a copy of this script to es01, es02, etc... then have each pointing to /data/es01/config, /data/es02/config, etc.. where the elasticsearch.yml file for each instance exists. This way each instance can be started with its own elasticsearch.yml file.

6

For this multi-instance setup, one thing that you need to look out for beside the HEAP_SIZE is the number of processors. You need to specifically set it per instance, otherwise, you will eventually run into problems related to the number of open files. I don't know about 2.2 but I've seen the problems at least with 2.1.x and 1.7.3.

For example, if the server physically has 16 cores, 64GB RAM, I normally reserve 50% RAM for Lucene, so I will have roughly 32GB left. In this case, I will run two instances with each has 30GB HEAP_SIZE and upto 8 cores (I would start with 4 cores) This way ES will behave properly.

7

thanks @thn @molsznski

i eventually did dockerize only elastic search and left kibana running as normal processes in multiple ports.

i came across with the multiple systemd/init scripts solution but i didn't went for it because you can configure ES_HEAP_SIZE only in /etc/sysconfig/elasticsearch, thus you cannot specify multiple elastic search instances with different heap size while having elastic search running as a service.
(please correct me if i am wrong)

that was the reason for deploying it with docker, because is a more standard way for other users to see how it has been deployed, with the mapping of configurations, directories, names, labels, tags to be just one command way.

1 Like
8

@nskalis I'm using CentOS 6, I was able to set ES_HEAP_SIZE right in /etc/init.d/elasticsearch or es01, es02, etc listed above. I did not touch /etc/sysconfig/elasticsearch.