ES Service not loading ES_PATH_CONF or CONF_DIR

[lzukel (Lance)]

ES Version Installed 6.1.2
Machine - RHEL 7.5

FILE: es-data0.service

        [Unit]
         Description=Elasticsearch
         Documentation=http://www.elastic.co
         Wants=network-online.target
         After=network-online.target
         
        [Service]
        RuntimeDirectory=elasticsearch
        Environment=ES_HOME=/usr/share/elasticsearch
        #Environment=CONF_DIR=/etc/elasticsearch
        Environment=ES_PATH_CONF=/etc/elasticsearch/es-data0
        #Environment=DATA_DIR=/var/lib/elasticsearch
        #Environment=LOG_DIR=/var/log/elasticsearch
        Environment=PID_DIR=/var/run/elasticsearch/es-data0
        EnvironmentFile=-/etc/sysconfig/es-data0
        
        WorkingDirectory=/usr/share/elasticsearch
        
        User=elasticsearch
        Group=elasticsearch
        
        #ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec
        
        ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet
                                                    #\
                                                    #-Edefault.path.logs=${LOG_DIR} \
                                                    #-Edefault.path.data=${DATA_DIR} \
                                                    #-Edefault.path.conf=${CONF_DIR}
         
        # StandardOutput is configured to redirect to journalctl since
        # some error messages may be logged in standard output before 
        # elasticsearch logging system is initialized. Elasticsearch
        # stores its logs in /var/log/elasticsearch and does not use
        # journalctl by default. If you also want to enable journalctl
        # logging, you can simply remove the "quiet" option from ExecStart.
        StandardOutput=journal
        StandardError=inherit
        
        # Specifies the maximum file descriptor number that can be opened by this process
        LimitNOFILE=65536
        
        # Specifies the maximum number of processes
        LimitNPROC=4096
        
        # Specifies the maximum number of bytes of memory that may be locked into RAM
        # Set to "infinity" if you use the 'bootstrap.memory_lock: true' option
        # in elasticsearch.yml and 'MAX_LOCKED_MEMORY=unlimited' in /etc/sysconfig/es-data0
        #LimitMEMLOCK=infinity
        
        # Specifies the maximum size of virtual memory
        LimitAS=infinity
        
        # Specifies the maximum file size
        LimitFSIZE=infinity
        
        # Disable timeout logic and wait until process is stopped
        TimeoutStopSec=0
        
        # SIGTERM signal is used to stop the Java process
        KillSignal=SIGTERM
        
        # Send the signal only to the JVM rather than its control group
        KillMode=process
        
        # Java process is never killed
        SendSIGKILL=no
        
        # When a JVM receives a SIGTERM signal it exits with code 143
        SuccessExitStatus=143
        
        [Install]
        WantedBy=multi-user.target
        
        # Built for distribution-6.1.2 (distribution)

FILE: /etc/sysconfig/es-data0

        DATA_DIR=/data/elasticsearch/es-data0
        ES_HOME=/usr/share/elasticsearch
        ES_JVM_OPTIONS=/etc/elasticsearch/es-data0/jvm.options
        ES_PATH_CONF=/etc/elasticsearch/es-data0
        ES_STARTUP_SLEEP_TIME=5
        LOG_DIR=/var/log/elasticsearch/es-data0
        MAX_LOCKED_MEMORY=unlimited
        MAX_MAP_COUNT=262144
        MAX_OPEN_FILES=65536
        PID_DIR=/var/run/elasticsearch
        RESTART_ON_UPGRADE=false

Error when starting any ES instance:

Config directory is /etc/elasticsearch/, from there the key- and truststore files are resolved relatively
...some stacktrace with irrelevant information...
Caused by: org.elasticsearch.ElasticsearchException: Unable to read /etc/elasticsearch/server.jks (/etc/elasticsearch/server.jks). Please make sure this files exists and is readable regarding to permissions. Property: searchguard.ssl.transport.keystore_filepath

The server.jks file is relative to the config/ directory
The server.jks file is present in the /etc/elasticsearch/es-data0 directory

Note: This is all managed by chef. This is based on the elasticsearch chef-cookbook
I have run diffs on all relevant files on a working machine that is loading the correct config directory from the environment variables set in the es-data0.service file

Startup from good running instance:

Config directory is /etc/elasticsearch/es-data0/, from there the key- and truststore files are resolved relatively

I need help getting ES to use the correct config directory.

[lzukel (Lance)]

I have also tried:

 /usr/share/elasticsearch/bin/elasticsearch -Epath.conf=/etc/elasticsearch/es-master/elasticsearch.yml 

Which still returns:

ES Config path is /etc/elasticsearch

on another node the same command returns:

ElasticsearchParseException[null-valued setting found for key [node.attr.rack_id] found at line number [46], column number [19]]

So I try passing:

/usr/share/elasticsearch/bin/elasticsearch -Epath.conf=/etc/elasticsearch/es-master/elasticsearch.yml -Enode.attr.rack_id=r1

Which still returns:

ElasticsearchParseException[null-valued setting found for key [node.attr.rack_id] found at line number [46], column number [19]]

This output only helps to bolster my conclusion that ES is not using any environment variables I specify

[jasontedor]

I am sorry for the troubles here, and I am committed to making this better based on the feedback we have seen on this issue.

The problem is ultimately that /etc/sysconfig/elasticsearch is sourced from elasticsearch-env. You need to edit /etc/sysconfig/elasticsearch which has a default value of ES_PATH_CONF for /etc/elasticsearch. I think you should be able to get away with removing these values from /etc/sysconfig/elasticsearch since you're setting them elsewhere, but in the worst case scenario you can duplicate the values there too.