Custom log4j2.propoerties for Elasticsearch in docker

Hello,

On docker, log messages go to the console and are handled by the configured Docker logging driver. To access logs we need to run docker logs $container_name.

I would like to change this behavior, and ask Elasticsearch to save logs into a specific path (same way if installed using RPM package for example)

For this i mounted the config folder including log4j2.properties with the same config that come from the RPM or TAR archive, but still not able to get logs into files in the specific path.

Any help please on this ?

version: '3.9'
services:
  es01:
    ...
    restart: always   
    volumes:
      - './config:/usr/share/elasticsearch/config:rw'
      - 'es_certs:/usr/share/elasticsearch/config/certificates:ro'
      - 'es_snapshot:/usr/share/elasticsearch/snapshot:rw'
      - 'es_logs:/usr/share/elasticsearch/logs:rw'
      - 'es_data:/usr/share/elasticsearch/data:rw'

This is the output of $ sudo docker logs es01
It seems that the docker image is ignoring my custom log4j2.properties ?

[0.004s][warning][logging] Output options for existing outputs are ignored.
[2022-07-21T16:27:41,793][INFO ][o.e.n.Node               ] [es01] version[8.3.2], pid[70], build[docker/8b0b1f23fbebecc3c88e4464319dea8989f374fd/2022-07-06T15:15:15.901688194Z], OS[Linux/3.10.0-1160.71.1.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/18.0.1.1/18.0.1.1+2-6]
[2022-07-21T16:27:41,808][INFO ][o.e.n.Node               ] [es01] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2022-07-21T16:27:41,808][INFO ][o.e.n.Node               ] [es01] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -Des.cgroups.hierarchy.override=/, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-2879924629542642311, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms1g, -Xmx1g, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-2879924629542642311, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.distribution.type=docker, --module-path=/usr/share/elasticsearch/lib, -Djdk.module.main=org.elasticsearch.server]
[2022-07-21T16:27:44,849][INFO ][c.a.c.i.j.JacksonVersion ] [es01] Package versions: jackson-annotations=2.13.2, jackson-core=2.13.2, jackson-databind=2.13.2.2, jackson-dataformat-xml=2.13.2, jackson-datatype-jsr310=2.13.2, azure-core=1.27.0, Troubleshooting version conflicts: https://aka.ms/azsdk/java/dependency/troubleshoot
[2022-07-21T16:27:46,760][INFO ][o.e.p.PluginsService     ] [es01] loaded module [aggs-matrix-stats]
[2022-07-21T16:27:46,760][INFO ][o.e.p.PluginsService     ] [es01] loaded module [analysis-common]
[2022-07-21T16:27:46,761][INFO ][o.e.p.PluginsService     ] [es01] loaded module [constant-keyword]
[2022-07-21T16:27:46,761][INFO ][o.e.p.PluginsService     ] [es01] loaded module [data-streams]
[2022-07-21T16:27:46,762][INFO ][o.e.p.PluginsService     ] [es01] loaded module [frozen-indices]
[2022-07-21T16:27:46,762][INFO ][o.e.p.PluginsService     ] [es01] loaded module [ingest-common]
[2022-07-21T16:27:46,762][INFO ][o.e.p.PluginsService     ] [es01] loaded module [ingest-geoip]
[2022-07-21T16:27:46,763][INFO ][o.e.p.PluginsService     ] [es01] loaded module [ingest-user-agent]
[2022-07-21T16:27:46,763][INFO ][o.e.p.PluginsService     ] [es01] loaded module [kibana]
[2022-07-21T16:27:46,763][INFO ][o.e.p.PluginsService     ] [es01] loaded module [lang-expression]
[2022-07-21T16:27:46,764][INFO ][o.e.p.PluginsService     ] [es01] loaded module [lang-mustache]
[2022-07-21T16:27:46,764][INFO ][o.e.p.PluginsService     ] [es01] loaded module [lang-painless]
[2022-07-21T16:27:46,764][INFO ][o.e.p.PluginsService     ] [es01] loaded module [legacy-geo]
[2022-07-21T16:27:46,765][INFO ][o.e.p.PluginsService     ] [es01] loaded module [mapper-extras]
[2022-07-21T16:27:46,765][INFO ][o.e.p.PluginsService     ] [es01] loaded module [mapper-version]
[2022-07-21T16:27:46,766][INFO ][o.e.p.PluginsService     ] [es01] loaded module [old-lucene-versions]
[2022-07-21T16:27:46,766][INFO ][o.e.p.PluginsService     ] [es01] loaded module [parent-join]
[2022-07-21T16:27:46,766][INFO ][o.e.p.PluginsService     ] [es01] loaded module [percolator]
[2022-07-21T16:27:46,767][INFO ][o.e.p.PluginsService     ] [es01] loaded module [rank-eval]
[2022-07-21T16:27:46,767][INFO ][o.e.p.PluginsService     ] [es01] loaded module [reindex]
[2022-07-21T16:27:46,767][INFO ][o.e.p.PluginsService     ] [es01] loaded module [repositories-metering-api]
[2022-07-21T16:27:46,768][INFO ][o.e.p.PluginsService     ] [es01] loaded module [repository-azure]
[2022-07-21T16:27:46,768][INFO ][o.e.p.PluginsService     ] [es01] loaded module [repository-encrypted]
[2022-07-21T16:27:46,768][INFO ][o.e.p.PluginsService     ] [es01] loaded module [repository-gcs]
[2022-07-21T16:27:46,769][INFO ][o.e.p.PluginsService     ] [es01] loaded module [repository-s3]
[2022-07-21T16:27:46,769][INFO ][o.e.p.PluginsService     ] [es01] loaded module [repository-url]
[2022-07-21T16:27:46,769][INFO ][o.e.p.PluginsService     ] [es01] loaded module [runtime-fields-common]
[2022-07-21T16:27:46,770][INFO ][o.e.p.PluginsService     ] [es01] loaded module [search-business-rules]
[2022-07-21T16:27:46,770][INFO ][o.e.p.PluginsService     ] [es01] loaded module [searchable-snapshots]
[2022-07-21T16:27:46,771][INFO ][o.e.p.PluginsService     ] [es01] loaded module [snapshot-based-recoveries]
[2022-07-21T16:27:46,771][INFO ][o.e.p.PluginsService     ] [es01] loaded module [snapshot-repo-test-kit]
[2022-07-21T16:27:46,771][INFO ][o.e.p.PluginsService     ] [es01] loaded module [spatial]
[2022-07-21T16:27:46,772][INFO ][o.e.p.PluginsService     ] [es01] loaded module [transform]
[2022-07-21T16:27:46,772][INFO ][o.e.p.PluginsService     ] [es01] loaded module [transport-netty4]
[2022-07-21T16:27:46,780][INFO ][o.e.p.PluginsService     ] [es01] loaded module [unsigned-long]
[2022-07-21T16:27:46,780][INFO ][o.e.p.PluginsService     ] [es01] loaded module [vector-tile]
[2022-07-21T16:27:46,781][INFO ][o.e.p.PluginsService     ] [es01] loaded module [vectors]
[2022-07-21T16:27:46,781][INFO ][o.e.p.PluginsService     ] [es01] loaded module [wildcard]
[2022-07-21T16:27:46,782][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-aggregate-metric]
[2022-07-21T16:27:46,782][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-analytics]
[2022-07-21T16:27:46,783][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-async]
[2022-07-21T16:27:46,783][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-async-search]
[2022-07-21T16:27:46,783][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-autoscaling]
[2022-07-21T16:27:46,784][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-ccr]
[2022-07-21T16:27:46,784][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-core]
[2022-07-21T16:27:46,784][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-deprecation]
[2022-07-21T16:27:46,784][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-enrich]
[2022-07-21T16:27:46,785][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-eql]
[2022-07-21T16:27:46,785][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-fleet]
[2022-07-21T16:27:46,785][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-graph]
[2022-07-21T16:27:46,786][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-identity-provider]
[2022-07-21T16:27:46,786][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-ilm]
[2022-07-21T16:27:46,786][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-logstash]
[2022-07-21T16:27:46,786][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-ml]
[2022-07-21T16:27:46,787][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-monitoring]
[2022-07-21T16:27:46,787][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-ql]
[2022-07-21T16:27:46,788][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-rollup]
[2022-07-21T16:27:46,788][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-security]
[2022-07-21T16:27:46,788][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-shutdown]
[2022-07-21T16:27:46,789][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-sql]
[2022-07-21T16:27:46,789][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-stack]
[2022-07-21T16:27:46,789][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-text-structure]
[2022-07-21T16:27:46,790][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-voting-only-node]
[2022-07-21T16:27:46,790][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-watcher]
[2022-07-21T16:27:46,790][INFO ][o.e.p.PluginsService     ] [es01] loaded plugin [mapper-murmur3]
[2022-07-21T16:27:46,791][INFO ][o.e.p.PluginsService     ] [es01] loaded plugin [mapper-size]
[2022-07-21T16:27:51,044][INFO ][o.e.e.NodeEnvironment    ] [es01] using [1] data paths, mounts [[/usr/share/elasticsearch/data_01 (/dev/mapper/centos-root)]], net usable_space [23.7gb], net total_space [47.7gb], types [xfs]
[2022-07-21T16:27:51,045][INFO ][o.e.e.NodeEnvironment    ] [es01] heap size [1gb], compressed ordinary object pointers [true]
[2022-07-21T16:27:51,135][INFO ][o.e.n.Node               ] [es01] node name [es01], node ID [9Q8W5kZ2RYGXwuw1MVteGQ], cluster name [dc02_es_cluster], roles [ml, data_hot, transform, data_content, data_warm, master, remote_cluster_client, data, data_cold, ingest, data_frozen]
[2022-07-21T16:27:55,902][INFO ][o.e.x.s.Security         ] [es01] Security is enabled
[2022-07-21T16:27:56,565][INFO ][o.e.x.s.a.s.FileRolesStore] [es01] parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]
[2022-07-21T16:27:57,185][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [es01] [controller/104] [Main.cc@123] controller (64 bit): Version 8.3.2 (Build c86b7174f20c42) Copyright (c) 2022 Elasticsearch BV
[2022-07-21T16:27:57,873][INFO ][o.e.t.n.NettyAllocator   ] [es01] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=1mb, factors={es.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=4mb, heap_size=1gb}]
[2022-07-21T16:27:57,912][INFO ][o.e.i.r.RecoverySettings ] [es01] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]
[2022-07-21T16:27:57,962][INFO ][o.e.d.DiscoveryModule    ] [es01] using discovery type [multi-node] and seed hosts providers [settings]

Log messages go to the console and are handled by the configured Docker logging driver. By default you can access logs with docker logs. If you would prefer the Elasticsearch container to write logs to disk, set the ES_LOG_STYLE environment variable to file. This causes Elasticsearch to use the same logging configuration as other Elasticsearch distribution formats.

More details here

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.