ES 5.2 log files not being written

[kwstx14 (Keith Stone)]

Logging is not working for us on Elasticsearch 5.2.0, and I can't find any documentation on the web that addresses the issue. Salient points:

log4j2.properties makes use of a couple of "exposed" variables that weren't in earlier versions of the product. An example line from the file that makes use of sys:es.logs.base_path and sys:es.logs.cluster_name is

appender.rolling.fileName = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}.log

The closest place I could find values that would be of use to log4j2 is in elasticsearch.yml, which defines both a log file path and a cluster name:

cluster.name: va_cluster_DDC_Dev
path.data: "/data/elasticsearch/es-01"
path.logs: "/data/elasticsearch/logs/es-01"

Log files should show up in /data/elasticsearch/logs/es-01, but they don't. But something strange I found is in /usr/share/elasticsearch:

[root@d03y07vemda0004 elasticsearch]# cd /usr/share/elasticsearch
[root@d03y07vemda0004 elasticsearch]# ls -lah
total 52K
drwxr-xr-x 9 elasticsearch elasticsearch 4.0K Nov 22 2017 .
drwxr-xr-x. 164 root root 4.0K Nov 22 2017 ..
drwxr-xr-x 2 root root 145 Nov 22 2017 bin
drwxr-xr-x 2 root root 4.0K Nov 22 2017 lib
-rw-r--r-- 1 root root 12K Jan 24 2017 LICENSE.txt
drwxr-xr-x 12 root root 4.0K Nov 22 2017 modules
-rw-r--r-- 1 root root 150 Jan 24 2017 NOTICE.txt
drwxr-xr-x 2 elasticsearch elasticsearch 6 Nov 22 2017 plugins
-rw-r--r-- 1 root root 8.9K Jan 24 2017 README.textile
drwxr-xr-x 2 elasticsearch elasticsearch 6 Nov 22 2017 scripts
drwxr-xr-x 2 elasticsearch elasticsearch 4.0K Nov 8 13:31 ${sys:es.logs.base_path}
drwxr-xr-x 2 elasticsearch elasticsearch 6 Nov 22 2017 templates_import

ls -lah './${sys:es.logs.base_path}'
total 40K
drwxr-xr-x 2 elasticsearch elasticsearch 4.0K Nov 8 13:31 .
drwxr-xr-x 9 elasticsearch elasticsearch 4.0K Nov 22 2017 ..
-rw-r--r-- 1 elasticsearch elasticsearch 3.8K Apr 16 2018 ${sys:es.logs.cluster_name}-2018-03-10-1.log.gz
-rw-r--r-- 1 elasticsearch elasticsearch 3.4K Sep 24 09:06 ${sys:es.logs.cluster_name}-2018-09-23-1.log.gz
-rw-r--r-- 1 elasticsearch elasticsearch 11K Nov 8 13:31 ${sys:es.logs.cluster_name}-2018-10-20-1.log.gz
-rw-r--r-- 1 elasticsearch elasticsearch 804 Nov 8 13:31 ${sys:es.logs.cluster_name}_deprecation.log
-rw-r--r-- 1 elasticsearch elasticsearch 0 Nov 22 2017 ${sys:es.logs.cluster_name}_index_indexing_slowlog.log
-rw-r--r-- 1 elasticsearch elasticsearch 0 Nov 22 2017 ${sys:es.logs.cluster_name}_index_search_slowlog.log
-rw-r--r-- 1 elasticsearch elasticsearch 6.2K Nov 8 13:33 ${sys:es.logs.cluster_name}.log

Directories for log files have been created in a location other than specified in elasticsearch.yml, and the directory names created there treat those values in log4j2.properties as string literals and creating directories with those variables as the names!

If not from elasticsearch.yml, where does log4j2 get its values for the log directory and cluster name? I'm ending up with what's essentially nonsense, and what's been put into place isn't working. I have looked for days for documentation or forums that might explain how log4j2 gets data for its logging specs from Elasticsearch, and I've found nothing.

[kwstx14 (Keith Stone)]

An update: I decided to try using the log4j2.properties from an earlier version of Elasticsearch (1.7.3). That file uses {sys:es.logs} instead of {sys:es.logs.base_path}{sys:file.separator}{sys:es.logs.cluster_name}. I restarted the Elasticsearch service, and it worked! I got log files in the intended location.

[kwstx14 (Keith Stone)]

One more update: I used the log4j2.properties file from Elastiscsearch 1.7.3 on version 5.2.0, and logging ran.

This turns out to be about the version of the elastic/elasticsearch Puppet module used. We're using release 6.0.0, which is the first version of the module that changed the variables used in log4j2.properties. It seems that I might want to consider backing off to release 5.5.0, which has the right variables in use in that erb file.