Running with Supervisor via command line

(Mike Sparr) #1

I would like to start/run ES nodes using Supervisor on Ubuntu 14.04.2 LTS. Using ES 1.7.1 and planning upgrade to 2.x when complete. Currently we use the package install and ES installed in /usr/share/elasticsearch/bin/ and config in /etc/elasticsearch/elasticsearch.yml. There is also a logging.yml file in conf directory.

We have a 7-server stack with 2 app, 2 db (mysql + redis), 3 ES nodes. To allow non-dev staff to restart/monitor services we are moving all apps and server systems under supervisord on every box, and then using a central admin app to monitor/manage/tail them all. We have nginx, cassandra, mysql, redis, and other node and python apps all running under supervisor and ES is final piece (before other ELK stack items).

I try to run ES using /usr/share/elasticsearch/bin/elasticsearch and get log4j errors. I try adding the Des.config=/etc/elasticsearch/elasticsearch.yml and same errors. I see older posts about using a -f flag but when I tried that it said invalid flag.

Can you provide the recommended practice to start ES from command line?

Supervisor config?

; Program definitions - one per invocation

command=/usr/share/elasticsearch/bin/elasticsearch -Des.config=/etc/elasticsearch/elasticsearch.yml


ubuntu@search1:/$ sudo /usr/share/elasticsearch/bin/elasticsearch 

Failed to configure logging...
org.elasticsearch.ElasticsearchException: Failed to load logging configuration
    at org.elasticsearch.common.logging.log4j.LogConfigurator.resolveConfig(
    at org.elasticsearch.common.logging.log4j.LogConfigurator.configure(
    at org.elasticsearch.bootstrap.Bootstrap.setupLogging(
    at org.elasticsearch.bootstrap.Bootstrap.main(
    at org.elasticsearch.bootstrap.Elasticsearch.main(
Caused by: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/config
    at sun.nio.fs.UnixException.translateToIOException(
    at sun.nio.fs.UnixException.rethrowAsIOException(
    at sun.nio.fs.UnixException.rethrowAsIOException(
    at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(
    at sun.nio.fs.UnixFileSystemProvider.readAttributes(
    at sun.nio.fs.LinuxFileSystemProvider.readAttributes(
    at java.nio.file.Files.readAttributes(
    at java.nio.file.FileTreeWalker.walk(
    at java.nio.file.FileTreeWalker.walk(
    at java.nio.file.Files.walkFileTree(
    at org.elasticsearch.common.logging.log4j.LogConfigurator.resolveConfig(
    ... 4 more
log4j:WARN No appenders could be found for logger (node).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See for more info.

(Magnus Bäck) #2

Try to mimic what the init script does. On my Debian-based system the process eventually ends up being invoked like this:

/usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Delasticsearch -Des.pidfile=/var/run/elasticsearch/ -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.7.1.jar:/usr/share/elasticsearch/lib/:/usr/share/elasticsearch/lib/sigar/ -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch

These arguments are partly supplied by /usr/share/elasticsearch/bin/elasticsearch and partly by /etc/init.d/elasticsearch. I think the immediate you problem you have ("Failed to configure logging") can be fixed by passing -Des.default.path.conf=/etc/elasticsearch.

(Mike Sparr) #3

Thanks. I'm working through it and figured if I symlink the config files in /etc/elasticsearch/ that the startup errors don't fail. I'll review what the init script runs and add more as needed.

This fixed errors:

ubuntu@search1:/usr/share/elasticsearch$ sudo mkdir config
ubuntu@search1:/usr/share/elasticsearch$ cd config
ubuntu@search1:/usr/share/elasticsearch/config$ sudo ln -s /etc/elasticsearch/elasticsearch.yml 
ubuntu@search1:/usr/share/elasticsearch/config$ sudo ln -s /etc/elasticsearch/logging.yml 

Started up and seems to be using config automatically after placing in that dir:

ubuntu@search1:/usr/share/elasticsearch/config$ cd ..
ubuntu@search1:/usr/share/elasticsearch$ cd bin/
ubuntu@search1:/usr/share/elasticsearch/bin$ sudo ./elasticsearch
[2015-09-07 21:59:01,257][INFO ][node                     ] [search1] version[1.7.1], pid[26675], build[b88f43f/2015-07-29T09:54:16Z]
[2015-09-07 21:59:01,258][INFO ][node                     ] [search1] initializing ...
[2015-09-07 21:59:01,377][INFO ][plugins                  ] [search1] loaded [cloud-aws, action-updatebyquery], sites [bigdesk, paramedic, HQ, head]
[2015-09-07 21:59:01,423][INFO ][env                      ] [search1] using [1] data paths, mounts [[/ (/dev/sda7)]], net usable_space [408.1gb], net total_space [432.1gb], types [ext4]
[2015-09-07 21:59:03,668][INFO ][node                     ] [search1] initialized
[2015-09-07 21:59:03,668][INFO ][node                     ] [search1] starting ...
[2015-09-07 21:59:03,725][INFO ][transport                ] [search1] bound_address {inet[/]}, publish_address {inet[search1/]}
[2015-09-07 21:59:03,733][INFO ][discovery                ] [search1] goomzee-prod/46Xf8IK1RZGylAqtXQCCUg
[2015-09-07 21:59:06,808][INFO ][cluster.service          ] [search1] detected_master [search3][GZ7CS2-wTqiAwEFDym8fzw][search3][inet[/]], added {[search3][GZ7CS2-wTqiAwEFDym8fzw][search3][inet[/]],[search2][hNPyUEwoQ46uD0NojjHAuQ][search2][inet[/]],}, reason: zen-disco-receive(from master [[search3][GZ7CS2-wTqiAwEFDym8fzw][search3][inet[/]]])
[2015-09-07 21:59:08,522][INFO ][http                     ] [search1] bound_address {inet[/]}, publish_address {inet[search1/]}
[2015-09-07 21:59:08,523][INFO ][node                     ] [search1] started

Now I have to update Ansible playbooks that install ES to add the symlink and update Supervisor conf. Fingers crossed. :wink:

(Mike Sparr) #4

Taking @magnusbaeck suggestion to simulate the init script, I used trial/error to execute and the following works in command line and with supervisord config:

Supervisor config in /etc/supervisor/conf.d/elasticsearch.conf (note vars replaced by Ansible template)

; Program definitions - one per invocation

command={{ es_home }}/elasticsearch -Des.default.path.conf={{ es_conf_dir }} -Des.default.config={{ es_conf_file }} -Des.default.path.logs={{ es_log_dir }}{{ es_data_dir }}{{ es_work_dir }}
environment=ES_HEAP_SIZE="{{ ansible_memtotal_mb // 2 }}m"

(system) #5