Log4j No appenders could be found for logger (bootstrap)

Elastic Stack Elasticsearch
1

HI,

I have 3 servers. And each server host 4 nodes ES v2.2.

Only 2 nodes can generate logs. The others have the following error :

Apr 10 14:12:48 elk-uat-m03 elasticsearch[2168]: log4j:WARN No appenders could be found for logger (bootstrap).
Apr 10 14:12:48 elk-uat-m03 elasticsearch[2168]: log4j:WARN Please initialize the log4j system properly.
Apr 10 14:12:48 elk-uat-m03 elasticsearch[2168]: log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

All my node share the same Home directory /usr/share/elasticsearch. Here is the contents of the directory :

drwxr-xr-x 3 root          root           4096 May 12  2016 bin
drwxr-xr-x 2 root          root           4096 Feb 12  2016 lib
-rw-r--r-- 1 root          root          11358 Jan 27  2016 LICENSE.txt
drwxr-xr-x 4 root          root           4096 Feb 12  2016 modules
-rw-r--r-- 1 root          root            150 Jan 27  2016 NOTICE.txt
drwxr-xr-x 9 elasticsearch elasticsearch  4096 May 12  2016 plugins
-rw-r--r-- 1 root          root           8700 Jan 27  2016 README.textile

All my node aslo share the same PID directory /var/run/elasticsearch

My instances are working correctly, some of them don't write in there log file.

I use the default logging.yml file for each node.

# you can override this using by setting a system property, for example -Des.logger.level=DEBUG
es.logger.level: INFO
rootLogger: ${es.logger.level}, console, file
logger:
  # log action execution errors for easier debugging
  action: DEBUG

  # deprecation logging, turn to DEBUG to see them
  deprecation: INFO, deprecation_log_file

  # reduce the logging for aws, too much is logged under the default INFO
  com.amazonaws: WARN
  # aws will try to do some sketchy JMX stuff, but its not needed.
  com.amazonaws.jmx.SdkMBeanRegistrySupport: ERROR
  com.amazonaws.metrics.AwsSdkMetrics: ERROR

  org.apache.http: INFO

  # gateway
  #gateway: DEBUG
  #index.gateway: DEBUG

  # peer shard recovery
  #indices.recovery: DEBUG

  # discovery
  #discovery: TRACE

  index.search.slowlog: TRACE, index_search_slow_log_file
  index.indexing.slowlog: TRACE, index_indexing_slow_log_file

additivity:
  index.search.slowlog: false
  index.indexing.slowlog: false
  deprecation: false

appender:
  console:
    type: console
    layout:
      type: consolePattern
      conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

  file:
    type: dailyRollingFile
    file: ${path.logs}/${cluster.name}.log
    datePattern: "'.'yyyy-MM-dd"
    layout:
      type: pattern
      conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %.10000m%n"

  # Use the following log4j-extras RollingFileAppender to enable gzip compression of log files. 
  # For more information see https://logging.apache.org/log4j/extras/apidocs/org/apache/log4j/rolling/RollingFileAppender.html
  #file:
    #type: extrasRollingFile
    #file: ${path.logs}/${cluster.name}.log
    #rollingPolicy: timeBased
    #rollingPolicy.FileNamePattern: ${path.logs}/${cluster.name}.log.%d{yyyy-MM-dd}.gz
    #layout:
      #type: pattern
      #conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

  deprecation_log_file:
    type: dailyRollingFile
    file: ${path.logs}/${cluster.name}_deprecation.log
    datePattern: "'.'yyyy-MM-dd"
    layout:
      type: pattern
      conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

  index_search_slow_log_file:
    type: dailyRollingFile
    file: ${path.logs}/${cluster.name}_index_search_slowlog.log
    datePattern: "'.'yyyy-MM-dd"
    layout:
      type: pattern
      conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

  index_indexing_slow_log_file:
    type: dailyRollingFile
    file: ${path.logs}/${cluster.name}_index_indexing_slowlog.log
    datePattern: "'.'yyyy-MM-dd"
    layout:
      type: pattern
      conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

All my configuration file are the same, so i don't know why some node are logging correctly and others have an Log4j error.

Thanks for your help.

2

Hey,

you need to check/diff your startup scripts and the configuration loaded there. I assume there might be leftovers from earlier package installations that may not have been overwritten. You can check the init script/systemd script and the /etc/elasticsearch directory.

--Alex

3

On each server, i have 4 conf directory, 4 Environment files and 4 init script systemd. One for each ES node.

For example on my server 1, my node01 running correctly, and the 3 others nodes don't write in their logging file.

I will show you the result of the diff commande between my node01 (client node) and my node02 (master node).

here is the diff between /usr/lib/systemd/system/elasticsearch-node01.service & /usr/lib/systemd/system/elasticsearch-node02.service

[root@elk-uat-m01 system]# diff elasticsearch-node01.service elasticsearch-node02.service 
9c9
< Environment=CONF_DIR=/etc/elasticsearch-node01
---
> Environment=CONF_DIR=/etc/elasticsearch-node02
11c11
< Environment=LOG_DIR=/var/log/elasticsearch-node01
---
> Environment=LOG_DIR=/var/log/elasticsearch-node02
13c13
< EnvironmentFile=-/etc/sysconfig/elasticsearch-node01
---
> EnvironmentFile=-/etc/sysconfig/elasticsearch-node02
23c23
<                                                 -Des.pidfile=${PID_DIR}/elasticsearch-node01.pid \
---
>                                                 -Des.pidfile=${PID_DIR}/elasticsearch-node02.pid \

here is the diff between /etc/elasticsearch-node01/elasticsearch.yml & /etc/elasticsearch-node02/elasticsearch.yml

<  node.name: uat-node01
---
>  node.name: uat-node02
38c38
<  node.master: false
---
>  node.master: true
51c51
<  path.logs: /var/log/elasticsearch-node01
---
>  path.logs: /var/log/elasticsearch-node02
72,74c72,74
<  network.host: 10.1.101.1
<  network.bind_host: 10.1.101.1
<  network.publish_host: 10.1.101.1
---
>  network.host: 10.1.101.2
>  network.bind_host: 10.1.101.2
>  network.publish_host: 10.1.101.2

The diff between /etc/elasticsearch-node01/logging.yml & /etc/elasticsearch-node02/logging.yml don't return nothing, so the files are the same.

And here is the diff between /etc/sysconfig/elasticsearch-node01& /etc/sysconfig/elasticsearch-node02

[root@elk-uat-m01 ~]# diff /etc/sysconfig/elasticsearch-node01 /etc/sysconfig/elasticsearch-node02 
9c9
< CONF_DIR=/etc/elasticsearch-node01
---
> CONF_DIR=/etc/elasticsearch-node02
15c15
< LOG_DIR=/var/log/elasticsearch-node01
---
> LOG_DIR=/var/log/elasticsearch-node02
22c22
< ES_HEAP_SIZE=6g
---
> ES_HEAP_SIZE=2g
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.