Custom log4j2.propoerties for Elasticsearch in docker

ylasri · July 21, 2022, 4:33pm

Hello,

On docker, log messages go to the console and are handled by the configured Docker logging driver. To access logs we need to run docker logs $container_name.

I would like to change this behavior, and ask Elasticsearch to save logs into a specific path (same way if installed using RPM package for example)

For this i mounted the config folder including log4j2.properties with the same config that come from the RPM or TAR archive, but still not able to get logs into files in the specific path.

Any help please on this ?

version: '3.9'
services:
  es01:
    ...
    restart: always   
    volumes:
      - './config:/usr/share/elasticsearch/config:rw'
      - 'es_certs:/usr/share/elasticsearch/config/certificates:ro'
      - 'es_snapshot:/usr/share/elasticsearch/snapshot:rw'
      - 'es_logs:/usr/share/elasticsearch/logs:rw'
      - 'es_data:/usr/share/elasticsearch/data:rw'

ylasri · July 21, 2022, 4:36pm

This is the output of $ sudo docker logs es01
It seems that the docker image is ignoring my custom log4j2.properties ?

[0.004s][warning][logging] Output options for existing outputs are ignored.
[2022-07-21T16:27:41,793][INFO ][o.e.n.Node               ] [es01] version[8.3.2], pid[70], build[docker/8b0b1f23fbebecc3c88e4464319dea8989f374fd/2022-07-06T15:15:15.901688194Z], OS[Linux/3.10.0-1160.71.1.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/18.0.1.1/18.0.1.1+2-6]
[2022-07-21T16:27:41,808][INFO ][o.e.n.Node               ] [es01] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2022-07-21T16:27:41,808][INFO ][o.e.n.Node               ] [es01] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -Des.cgroups.hierarchy.override=/, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-2879924629542642311, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms1g, -Xmx1g, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-2879924629542642311, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.distribution.type=docker, --module-path=/usr/share/elasticsearch/lib, -Djdk.module.main=org.elasticsearch.server]
[2022-07-21T16:27:44,849][INFO ][c.a.c.i.j.JacksonVersion ] [es01] Package versions: jackson-annotations=2.13.2, jackson-core=2.13.2, jackson-databind=2.13.2.2, jackson-dataformat-xml=2.13.2, jackson-datatype-jsr310=2.13.2, azure-core=1.27.0, Troubleshooting version conflicts: https://aka.ms/azsdk/java/dependency/troubleshoot
[2022-07-21T16:27:46,760][INFO ][o.e.p.PluginsService     ] [es01] loaded module [aggs-matrix-stats]
[2022-07-21T16:27:46,760][INFO ][o.e.p.PluginsService     ] [es01] loaded module [analysis-common]
[2022-07-21T16:27:46,761][INFO ][o.e.p.PluginsService     ] [es01] loaded module [constant-keyword]
[2022-07-21T16:27:46,761][INFO ][o.e.p.PluginsService     ] [es01] loaded module [data-streams]
[2022-07-21T16:27:46,762][INFO ][o.e.p.PluginsService     ] [es01] loaded module [frozen-indices]
[2022-07-21T16:27:46,762][INFO ][o.e.p.PluginsService     ] [es01] loaded module [ingest-common]
[2022-07-21T16:27:46,762][INFO ][o.e.p.PluginsService     ] [es01] loaded module [ingest-geoip]
[2022-07-21T16:27:46,763][INFO ][o.e.p.PluginsService     ] [es01] loaded module [ingest-user-agent]
[2022-07-21T16:27:46,763][INFO ][o.e.p.PluginsService     ] [es01] loaded module [kibana]
[2022-07-21T16:27:46,763][INFO ][o.e.p.PluginsService     ] [es01] loaded module [lang-expression]
[2022-07-21T16:27:46,764][INFO ][o.e.p.PluginsService     ] [es01] loaded module [lang-mustache]
[2022-07-21T16:27:46,764][INFO ][o.e.p.PluginsService     ] [es01] loaded module [lang-painless]
[2022-07-21T16:27:46,764][INFO ][o.e.p.PluginsService     ] [es01] loaded module [legacy-geo]
[2022-07-21T16:27:46,765][INFO ][o.e.p.PluginsService     ] [es01] loaded module [mapper-extras]
[2022-07-21T16:27:46,765][INFO ][o.e.p.PluginsService     ] [es01] loaded module [mapper-version]
[2022-07-21T16:27:46,766][INFO ][o.e.p.PluginsService     ] [es01] loaded module [old-lucene-versions]
[2022-07-21T16:27:46,766][INFO ][o.e.p.PluginsService     ] [es01] loaded module [parent-join]
[2022-07-21T16:27:46,766][INFO ][o.e.p.PluginsService     ] [es01] loaded module [percolator]
[2022-07-21T16:27:46,767][INFO ][o.e.p.PluginsService     ] [es01] loaded module [rank-eval]
[2022-07-21T16:27:46,767][INFO ][o.e.p.PluginsService     ] [es01] loaded module [reindex]
[2022-07-21T16:27:46,767][INFO ][o.e.p.PluginsService     ] [es01] loaded module [repositories-metering-api]
[2022-07-21T16:27:46,768][INFO ][o.e.p.PluginsService     ] [es01] loaded module [repository-azure]
[2022-07-21T16:27:46,768][INFO ][o.e.p.PluginsService     ] [es01] loaded module [repository-encrypted]
[2022-07-21T16:27:46,768][INFO ][o.e.p.PluginsService     ] [es01] loaded module [repository-gcs]
[2022-07-21T16:27:46,769][INFO ][o.e.p.PluginsService     ] [es01] loaded module [repository-s3]
[2022-07-21T16:27:46,769][INFO ][o.e.p.PluginsService     ] [es01] loaded module [repository-url]
[2022-07-21T16:27:46,769][INFO ][o.e.p.PluginsService     ] [es01] loaded module [runtime-fields-common]
[2022-07-21T16:27:46,770][INFO ][o.e.p.PluginsService     ] [es01] loaded module [search-business-rules]
[2022-07-21T16:27:46,770][INFO ][o.e.p.PluginsService     ] [es01] loaded module [searchable-snapshots]
[2022-07-21T16:27:46,771][INFO ][o.e.p.PluginsService     ] [es01] loaded module [snapshot-based-recoveries]
[2022-07-21T16:27:46,771][INFO ][o.e.p.PluginsService     ] [es01] loaded module [snapshot-repo-test-kit]
[2022-07-21T16:27:46,771][INFO ][o.e.p.PluginsService     ] [es01] loaded module [spatial]
[2022-07-21T16:27:46,772][INFO ][o.e.p.PluginsService     ] [es01] loaded module [transform]
[2022-07-21T16:27:46,772][INFO ][o.e.p.PluginsService     ] [es01] loaded module [transport-netty4]
[2022-07-21T16:27:46,780][INFO ][o.e.p.PluginsService     ] [es01] loaded module [unsigned-long]
[2022-07-21T16:27:46,780][INFO ][o.e.p.PluginsService     ] [es01] loaded module [vector-tile]
[2022-07-21T16:27:46,781][INFO ][o.e.p.PluginsService     ] [es01] loaded module [vectors]
[2022-07-21T16:27:46,781][INFO ][o.e.p.PluginsService     ] [es01] loaded module [wildcard]
[2022-07-21T16:27:46,782][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-aggregate-metric]
[2022-07-21T16:27:46,782][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-analytics]
[2022-07-21T16:27:46,783][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-async]
[2022-07-21T16:27:46,783][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-async-search]
[2022-07-21T16:27:46,783][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-autoscaling]
[2022-07-21T16:27:46,784][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-ccr]
[2022-07-21T16:27:46,784][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-core]
[2022-07-21T16:27:46,784][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-deprecation]
[2022-07-21T16:27:46,784][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-enrich]
[2022-07-21T16:27:46,785][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-eql]
[2022-07-21T16:27:46,785][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-fleet]
[2022-07-21T16:27:46,785][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-graph]
[2022-07-21T16:27:46,786][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-identity-provider]
[2022-07-21T16:27:46,786][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-ilm]
[2022-07-21T16:27:46,786][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-logstash]
[2022-07-21T16:27:46,786][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-ml]
[2022-07-21T16:27:46,787][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-monitoring]
[2022-07-21T16:27:46,787][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-ql]
[2022-07-21T16:27:46,788][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-rollup]
[2022-07-21T16:27:46,788][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-security]
[2022-07-21T16:27:46,788][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-shutdown]
[2022-07-21T16:27:46,789][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-sql]
[2022-07-21T16:27:46,789][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-stack]
[2022-07-21T16:27:46,789][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-text-structure]
[2022-07-21T16:27:46,790][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-voting-only-node]
[2022-07-21T16:27:46,790][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-watcher]
[2022-07-21T16:27:46,790][INFO ][o.e.p.PluginsService     ] [es01] loaded plugin [mapper-murmur3]
[2022-07-21T16:27:46,791][INFO ][o.e.p.PluginsService     ] [es01] loaded plugin [mapper-size]
[2022-07-21T16:27:51,044][INFO ][o.e.e.NodeEnvironment    ] [es01] using [1] data paths, mounts [[/usr/share/elasticsearch/data_01 (/dev/mapper/centos-root)]], net usable_space [23.7gb], net total_space [47.7gb], types [xfs]
[2022-07-21T16:27:51,045][INFO ][o.e.e.NodeEnvironment    ] [es01] heap size [1gb], compressed ordinary object pointers [true]
[2022-07-21T16:27:51,135][INFO ][o.e.n.Node               ] [es01] node name [es01], node ID [9Q8W5kZ2RYGXwuw1MVteGQ], cluster name [dc02_es_cluster], roles [ml, data_hot, transform, data_content, data_warm, master, remote_cluster_client, data, data_cold, ingest, data_frozen]
[2022-07-21T16:27:55,902][INFO ][o.e.x.s.Security         ] [es01] Security is enabled
[2022-07-21T16:27:56,565][INFO ][o.e.x.s.a.s.FileRolesStore] [es01] parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]
[2022-07-21T16:27:57,185][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [es01] [controller/104] [Main.cc@123] controller (64 bit): Version 8.3.2 (Build c86b7174f20c42) Copyright (c) 2022 Elasticsearch BV
[2022-07-21T16:27:57,873][INFO ][o.e.t.n.NettyAllocator   ] [es01] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=1mb, factors={es.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=4mb, heap_size=1gb}]
[2022-07-21T16:27:57,912][INFO ][o.e.i.r.RecoverySettings ] [es01] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]
[2022-07-21T16:27:57,962][INFO ][o.e.d.DiscoveryModule    ] [es01] using discovery type [multi-node] and seed hosts providers [settings]

ylasri · August 3, 2022, 11:38am

Log messages go to the console and are handled by the configured Docker logging driver. By default you can access logs with docker logs. If you would prefer the Elasticsearch container to write logs to disk, set the ES_LOG_STYLE environment variable to file. This causes Elasticsearch to use the same logging configuration as other Elasticsearch distribution formats.

ylasri · August 3, 2022, 11:38am

More details here

github.com/elastic/elasticsearch

Custom log4j2.propoerties for Elasticsearch in docker

opened 09:30AM - 28 Jul 22 UTC

ylasri

>bug :Delivery/Packaging team-discuss Team:Delivery

### Elasticsearch Version 8.3.2 ### Installed Plugins N/A ### Java Version rel="noopener" class="show-more">…_bundled_ are handled by the configured Docker logging driver. To access logs we need to run docker logs $container_name. Elasticsearch to save logs into a specific path (same way if installed using RPM package for example) log4j2.properties with the same config that come from the RPM or TAR archive, but still not able to get logs into files in the specific path. tificates:ro' t:rw' custom log4j2.properties existing outputs are ignored. ] [es01] version[8.3.2], pid[70], build[docker/8b0b1f23fbebecc3c88e4464319dea8989f374fd/2022-07-06T15:15:15.901688194Z], OS[Linux/3.10.0-1160.71.1.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/18.0.1.1/18.0.1.1+2-6] ] [es01] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true] ] [es01] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -Des.cgroups.hierarchy.override=/, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-2879924629542642311, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms1g, -Xmx1g, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-2879924629542642311, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.distribution.type=docker, --module-path=/usr/share/elasticsearch/lib, -Djdk.module.main=org.elasticsearch.server] ] [es01] Package versions: jackson-annotations=2.13.2, jackson-core=2.13.2, jackson-databind=2.13.2.2, jackson-dataformat-xml=2.13.2, jackson-datatype-jsr310=2.13.2, azure-core=1.27.0, Troubleshooting version conflicts: https://aka.ms/azsdk/java/dependency/troubleshoot ] [es01] loaded module [aggs-matrix-stats] ] [es01] loaded module [analysis-common] ] [es01] loaded module [constant-keyword] ] [es01] loaded module [data-streams] ] [es01] loaded module [frozen-indices] ] [es01] loaded module [ingest-common] ] [es01] loaded module [ingest-geoip] ] [es01] loaded module [ingest-user-agent] ] [es01] loaded module [kibana] ] [es01] loaded module [lang-expression] ] [es01] loaded module [lang-mustache] ] [es01] loaded module [lang-painless] ] [es01] loaded module [legacy-geo] ] [es01] loaded module [mapper-extras] ] [es01] loaded module [mapper-version] ] [es01] loaded module [old-lucene-versions] ] [es01] loaded module [parent-join] ] [es01] loaded module [percolator] ] [es01] loaded module [rank-eval] ] [es01] loaded module [reindex] ] [es01] loaded module [repositories-metering-api] ] [es01] loaded module [repository-azure] ] [es01] loaded module [repository-encrypted] ] [es01] loaded module [repository-gcs] ] [es01] loaded module [repository-s3] ] [es01] loaded module [repository-url] ] [es01] loaded module [runtime-fields-common] ] [es01] loaded module [search-business-rules] ] [es01] loaded module [searchable-snapshots] ] [es01] loaded module [snapshot-based-recoveries] ] [es01] loaded module [snapshot-repo-test-kit] ] [es01] loaded module [spatial] ] [es01] loaded module [transform] ] [es01] loaded module [transport-netty4] ] [es01] loaded module [unsigned-long] ] [es01] loaded module [vector-tile] ] [es01] loaded module [vectors] ] [es01] loaded module [wildcard] ] [es01] loaded module [x-pack-aggregate-metric] ] [es01] loaded module [x-pack-analytics] ] [es01] loaded module [x-pack-async] ] [es01] loaded module [x-pack-async-search] ] [es01] loaded module [x-pack-autoscaling] ] [es01] loaded module [x-pack-ccr] ] [es01] loaded module [x-pack-core] ] [es01] loaded module [x-pack-deprecation] ] [es01] loaded module [x-pack-enrich] ] [es01] loaded module [x-pack-eql] ] [es01] loaded module [x-pack-fleet] ] [es01] loaded module [x-pack-graph] ] [es01] loaded module [x-pack-identity-provider] ] [es01] loaded module [x-pack-ilm] ] [es01] loaded module [x-pack-logstash] ] [es01] loaded module [x-pack-ml] ] [es01] loaded module [x-pack-monitoring] ] [es01] loaded module [x-pack-ql] ] [es01] loaded module [x-pack-rollup] ] [es01] loaded module [x-pack-security] ] [es01] loaded module [x-pack-shutdown] ] [es01] loaded module [x-pack-sql] ] [es01] loaded module [x-pack-stack] ] [es01] loaded module [x-pack-text-structure] ] [es01] loaded module [x-pack-voting-only-node] ] [es01] loaded module [x-pack-watcher] ] [es01] loaded plugin [mapper-murmur3] ] [es01] loaded plugin [mapper-size] ] [es01] using [1] data paths, mounts [[/usr/share/elasticsearch/data_01 (/dev/mapper/centos-root)]], net usable_space [23.7gb], net total_space [47.7gb], types [xfs] ] [es01] heap size [1gb], compressed ordinary object pointers [true] ] [es01] node name [es01], node ID [9Q8W5kZ2RYGXwuw1MVteGQ], cluster name [dc02_es_cluster], roles [ml, data_hot, transform, data_content, data_warm, master, remote_cluster_client, data, data_cold, ingest, data_frozen] ] [es01] Security is enabled [es01] parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml] [es01] [controller/104] [Main.cc@123] controller (64 bit): Version 8.3.2 (Build c86b7174f20c42) Copyright (c) 2022 Elasticsearch BV ] [es01] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=1mb, factors={es.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=4mb, heap_size=1gb}] ] [es01] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b] ] [es01] using discovery type [multi-node] and seed hosts providers [settings] example the same as used for Linux archive so that logs can be redirected directly to a folder)

system · August 31, 2022, 11:39am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
One issue about elastic-docker Elasticsearch	6	442	April 26, 2018
Log custom plugin classes and read it with docker logs Elasticsearch	1	340	January 1, 2019
How to collect elasticsearch logs in docker Elasticsearch docker	3	18068	December 27, 2019
Docker-error: no log4j2.properties found; tried [/usr/share/elasticsearch/config] and its subdirectories Elasticsearch	2	4474	December 31, 2017
No Logs in Log Folder Elasticsearch	4	4562	October 8, 2017

Custom log4j2.propoerties for Elasticsearch in docker

Related topics