Running with Supervisor via command line

Elastic Stack Elasticsearch
1

I would like to start/run ES nodes using Supervisor on Ubuntu 14.04.2 LTS. Using ES 1.7.1 and planning upgrade to 2.x when complete. Currently we use the package install and ES installed in /usr/share/elasticsearch/bin/ and config in /etc/elasticsearch/elasticsearch.yml. There is also a logging.yml file in conf directory.

We have a 7-server stack with 2 app, 2 db (mysql + redis), 3 ES nodes. To allow non-dev staff to restart/monitor services we are moving all apps and server systems under supervisord on every box, and then using a central admin app to monitor/manage/tail them all. We have nginx, cassandra, mysql, redis, and other node and python apps all running under supervisor and ES is final piece (before other ELK stack items).

I try to run ES using /usr/share/elasticsearch/bin/elasticsearch and get log4j errors. I try adding the Des.config=/etc/elasticsearch/elasticsearch.yml and same errors. I see older posts about using a -f flag but when I tried that it said invalid flag.

Can you provide the recommended practice to start ES from command line?

Supervisor config?

; Program definitions - one per invocation

[program:elasticsearch]
command=/usr/share/elasticsearch/bin/elasticsearch -Des.config=/etc/elasticsearch/elasticsearch.yml
stdout_logfile=/var/log/supervisor/elasticsearch.log
redirect_stderr=true
autostart=true
;directory=
;environment=??
user=elasticsearch

Error:

ubuntu@search1:/$ sudo /usr/share/elasticsearch/bin/elasticsearch 

-Des.config=/etc/elasticsearch/elasticsearch.yml
Failed to configure logging...
org.elasticsearch.ElasticsearchException: Failed to load logging configuration
    at org.elasticsearch.common.logging.log4j.LogConfigurator.resolveConfig(LogConfigurator.java:142)
    at org.elasticsearch.common.logging.log4j.LogConfigurator.configure(LogConfigurator.java:92)
    at org.elasticsearch.bootstrap.Bootstrap.setupLogging(Bootstrap.java:131)
    at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:216)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)
Caused by: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/config
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
    at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
    at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:97)
    at java.nio.file.Files.readAttributes(Files.java:1686)
    at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:109)
    at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:69)
    at java.nio.file.Files.walkFileTree(Files.java:2602)
    at org.elasticsearch.common.logging.log4j.LogConfigurator.resolveConfig(LogConfigurator.java:126)
    ... 4 more
log4j:WARN No appenders could be found for logger (node).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
2

Try to mimic what the init script does. On my Debian-based system the process eventually ends up being invoked like this:

/usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Delasticsearch -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.7.1.jar:/usr/share/elasticsearch/lib/:/usr/share/elasticsearch/lib/sigar/ -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch

These arguments are partly supplied by /usr/share/elasticsearch/bin/elasticsearch and partly by /etc/init.d/elasticsearch. I think the immediate you problem you have ("Failed to configure logging") can be fixed by passing -Des.default.path.conf=/etc/elasticsearch.

3

Thanks. I'm working through it and figured if I symlink the config files in /etc/elasticsearch/ that the startup errors don't fail. I'll review what the init script runs and add more as needed.

This fixed errors:

ubuntu@search1:/usr/share/elasticsearch$ sudo mkdir config
ubuntu@search1:/usr/share/elasticsearch$ cd config
ubuntu@search1:/usr/share/elasticsearch/config$ sudo ln -s /etc/elasticsearch/elasticsearch.yml 
ubuntu@search1:/usr/share/elasticsearch/config$ sudo ln -s /etc/elasticsearch/logging.yml

Started up and seems to be using config automatically after placing in that dir:

ubuntu@search1:/usr/share/elasticsearch/config$ cd ..
ubuntu@search1:/usr/share/elasticsearch$ cd bin/
ubuntu@search1:/usr/share/elasticsearch/bin$ sudo ./elasticsearch
[2015-09-07 21:59:01,257][INFO ][node                     ] [search1] version[1.7.1], pid[26675], build[b88f43f/2015-07-29T09:54:16Z]
[2015-09-07 21:59:01,258][INFO ][node                     ] [search1] initializing ...
[2015-09-07 21:59:01,377][INFO ][plugins                  ] [search1] loaded [cloud-aws, action-updatebyquery], sites [bigdesk, paramedic, HQ, head]
[2015-09-07 21:59:01,423][INFO ][env                      ] [search1] using [1] data paths, mounts [[/ (/dev/sda7)]], net usable_space [408.1gb], net total_space [432.1gb], types [ext4]
[2015-09-07 21:59:03,668][INFO ][node                     ] [search1] initialized
[2015-09-07 21:59:03,668][INFO ][node                     ] [search1] starting ...
[2015-09-07 21:59:03,725][INFO ][transport                ] [search1] bound_address {inet[/10.0.1.5:9300]}, publish_address {inet[search1/10.0.1.5:9300]}
[2015-09-07 21:59:03,733][INFO ][discovery                ] [search1] goomzee-prod/46Xf8IK1RZGylAqtXQCCUg
[2015-09-07 21:59:06,808][INFO ][cluster.service          ] [search1] detected_master [search3][GZ7CS2-wTqiAwEFDym8fzw][search3][inet[/10.0.1.7:9300]], added {[search3][GZ7CS2-wTqiAwEFDym8fzw][search3][inet[/10.0.1.7:9300]],[search2][hNPyUEwoQ46uD0NojjHAuQ][search2][inet[/10.0.1.6:9300]],}, reason: zen-disco-receive(from master [[search3][GZ7CS2-wTqiAwEFDym8fzw][search3][inet[/10.0.1.7:9300]]])
[2015-09-07 21:59:08,522][INFO ][http                     ] [search1] bound_address {inet[/10.0.1.5:9200]}, publish_address {inet[search1/10.0.1.5:9200]}
[2015-09-07 21:59:08,523][INFO ][node                     ] [search1] started

Now I have to update Ansible playbooks that install ES to add the symlink and update Supervisor conf. Fingers crossed. :wink:

4

Taking @magnusbaeck suggestion to simulate the init script, I used trial/error to execute and the following works in command line and with supervisord config:

Supervisor config in /etc/supervisor/conf.d/elasticsearch.conf (note vars replaced by Ansible template)

; Program definitions - one per invocation

[program:elasticsearch]
command={{ es_home }}/elasticsearch -Des.default.path.conf={{ es_conf_dir }} -Des.default.config={{ es_conf_file }} -Des.default.path.logs={{ es_log_dir }} -Des.default.path.data={{ es_data_dir }} -Des.default.path.work={{ es_work_dir }}
stdout_logfile=/var/log/supervisor/elasticsearch.log
redirect_stderr=true
autostart=true
;directory=
environment=ES_HEAP_SIZE="{{ ansible_memtotal_mb // 2 }}m"
user=elasticsearch
1 Like