Custom log4j2.propoerties for Elasticsearch in docker

ylasri · July 21, 2022, 4:33pm

Hello,

On docker, log messages go to the console and are handled by the configured Docker logging driver. To access logs we need to run docker logs $container_name.

I would like to change this behavior, and ask Elasticsearch to save logs into a specific path (same way if installed using RPM package for example)

For this i mounted the config folder including log4j2.properties with the same config that come from the RPM or TAR archive, but still not able to get logs into files in the specific path.

Any help please on this ?

version: '3.9'
services:
  es01:
    ...
    restart: always   
    volumes:
      - './config:/usr/share/elasticsearch/config:rw'
      - 'es_certs:/usr/share/elasticsearch/config/certificates:ro'
      - 'es_snapshot:/usr/share/elasticsearch/snapshot:rw'
      - 'es_logs:/usr/share/elasticsearch/logs:rw'
      - 'es_data:/usr/share/elasticsearch/data:rw'

ylasri · July 21, 2022, 4:36pm

This is the output of $ sudo docker logs es01
It seems that the docker image is ignoring my custom log4j2.properties ?

[0.004s][warning][logging] Output options for existing outputs are ignored.
[2022-07-21T16:27:41,793][INFO ][o.e.n.Node               ] [es01] version[8.3.2], pid[70], build[docker/8b0b1f23fbebecc3c88e4464319dea8989f374fd/2022-07-06T15:15:15.901688194Z], OS[Linux/3.10.0-1160.71.1.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/18.0.1.1/18.0.1.1+2-6]
[2022-07-21T16:27:41,808][INFO ][o.e.n.Node               ] [es01] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2022-07-21T16:27:41,808][INFO ][o.e.n.Node               ] [es01] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -Des.cgroups.hierarchy.override=/, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-2879924629542642311, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms1g, -Xmx1g, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-2879924629542642311, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.distribution.type=docker, --module-path=/usr/share/elasticsearch/lib, -Djdk.module.main=org.elasticsearch.server]
[2022-07-21T16:27:44,849][INFO ][c.a.c.i.j.JacksonVersion ] [es01] Package versions: jackson-annotations=2.13.2, jackson-core=2.13.2, jackson-databind=2.13.2.2, jackson-dataformat-xml=2.13.2, jackson-datatype-jsr310=2.13.2, azure-core=1.27.0, Troubleshooting version conflicts: https://aka.ms/azsdk/java/dependency/troubleshoot
[2022-07-21T16:27:46,760][INFO ][o.e.p.PluginsService     ] [es01] loaded module [aggs-matrix-stats]
[2022-07-21T16:27:46,760][INFO ][o.e.p.PluginsService     ] [es01] loaded module [analysis-common]
[2022-07-21T16:27:46,761][INFO ][o.e.p.PluginsService     ] [es01] loaded module [constant-keyword]
[2022-07-21T16:27:46,761][INFO ][o.e.p.PluginsService     ] [es01] loaded module [data-streams]
[2022-07-21T16:27:46,762][INFO ][o.e.p.PluginsService     ] [es01] loaded module [frozen-indices]
[2022-07-21T16:27:46,762][INFO ][o.e.p.PluginsService     ] [es01] loaded module [ingest-common]
[2022-07-21T16:27:46,762][INFO ][o.e.p.PluginsService     ] [es01] loaded module [ingest-geoip]
[2022-07-21T16:27:46,763][INFO ][o.e.p.PluginsService     ] [es01] loaded module [ingest-user-agent]
[2022-07-21T16:27:46,763][INFO ][o.e.p.PluginsService     ] [es01] loaded module [kibana]
[2022-07-21T16:27:46,763][INFO ][o.e.p.PluginsService     ] [es01] loaded module [lang-expression]
[2022-07-21T16:27:46,764][INFO ][o.e.p.PluginsService     ] [es01] loaded module [lang-mustache]
[2022-07-21T16:27:46,764][INFO ][o.e.p.PluginsService     ] [es01] loaded module [lang-painless]
[2022-07-21T16:27:46,764][INFO ][o.e.p.PluginsService     ] [es01] loaded module [legacy-geo]
[2022-07-21T16:27:46,765][INFO ][o.e.p.PluginsService     ] [es01] loaded module [mapper-extras]
[2022-07-21T16:27:46,765][INFO ][o.e.p.PluginsService     ] [es01] loaded module [mapper-version]
[2022-07-21T16:27:46,766][INFO ][o.e.p.PluginsService     ] [es01] loaded module [old-lucene-versions]
[2022-07-21T16:27:46,766][INFO ][o.e.p.PluginsService     ] [es01] loaded module [parent-join]
[2022-07-21T16:27:46,766][INFO ][o.e.p.PluginsService     ] [es01] loaded module [percolator]
[2022-07-21T16:27:46,767][INFO ][o.e.p.PluginsService     ] [es01] loaded module [rank-eval]
[2022-07-21T16:27:46,767][INFO ][o.e.p.PluginsService     ] [es01] loaded module [reindex]
[2022-07-21T16:27:46,767][INFO ][o.e.p.PluginsService     ] [es01] loaded module [repositories-metering-api]
[2022-07-21T16:27:46,768][INFO ][o.e.p.PluginsService     ] [es01] loaded module [repository-azure]
[2022-07-21T16:27:46,768][INFO ][o.e.p.PluginsService     ] [es01] loaded module [repository-encrypted]
[2022-07-21T16:27:46,768][INFO ][o.e.p.PluginsService     ] [es01] loaded module [repository-gcs]
[2022-07-21T16:27:46,769][INFO ][o.e.p.PluginsService     ] [es01] loaded module [repository-s3]
[2022-07-21T16:27:46,769][INFO ][o.e.p.PluginsService     ] [es01] loaded module [repository-url]
[2022-07-21T16:27:46,769][INFO ][o.e.p.PluginsService     ] [es01] loaded module [runtime-fields-common]
[2022-07-21T16:27:46,770][INFO ][o.e.p.PluginsService     ] [es01] loaded module [search-business-rules]
[2022-07-21T16:27:46,770][INFO ][o.e.p.PluginsService     ] [es01] loaded module [searchable-snapshots]
[2022-07-21T16:27:46,771][INFO ][o.e.p.PluginsService     ] [es01] loaded module [snapshot-based-recoveries]
[2022-07-21T16:27:46,771][INFO ][o.e.p.PluginsService     ] [es01] loaded module [snapshot-repo-test-kit]
[2022-07-21T16:27:46,771][INFO ][o.e.p.PluginsService     ] [es01] loaded module [spatial]
[2022-07-21T16:27:46,772][INFO ][o.e.p.PluginsService     ] [es01] loaded module [transform]
[2022-07-21T16:27:46,772][INFO ][o.e.p.PluginsService     ] [es01] loaded module [transport-netty4]
[2022-07-21T16:27:46,780][INFO ][o.e.p.PluginsService     ] [es01] loaded module [unsigned-long]
[2022-07-21T16:27:46,780][INFO ][o.e.p.PluginsService     ] [es01] loaded module [vector-tile]
[2022-07-21T16:27:46,781][INFO ][o.e.p.PluginsService     ] [es01] loaded module [vectors]
[2022-07-21T16:27:46,781][INFO ][o.e.p.PluginsService     ] [es01] loaded module [wildcard]
[2022-07-21T16:27:46,782][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-aggregate-metric]
[2022-07-21T16:27:46,782][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-analytics]
[2022-07-21T16:27:46,783][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-async]
[2022-07-21T16:27:46,783][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-async-search]
[2022-07-21T16:27:46,783][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-autoscaling]
[2022-07-21T16:27:46,784][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-ccr]
[2022-07-21T16:27:46,784][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-core]
[2022-07-21T16:27:46,784][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-deprecation]
[2022-07-21T16:27:46,784][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-enrich]
[2022-07-21T16:27:46,785][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-eql]
[2022-07-21T16:27:46,785][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-fleet]
[2022-07-21T16:27:46,785][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-graph]
[2022-07-21T16:27:46,786][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-identity-provider]
[2022-07-21T16:27:46,786][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-ilm]
[2022-07-21T16:27:46,786][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-logstash]
[2022-07-21T16:27:46,786][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-ml]
[2022-07-21T16:27:46,787][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-monitoring]
[2022-07-21T16:27:46,787][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-ql]
[2022-07-21T16:27:46,788][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-rollup]
[2022-07-21T16:27:46,788][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-security]
[2022-07-21T16:27:46,788][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-shutdown]
[2022-07-21T16:27:46,789][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-sql]
[2022-07-21T16:27:46,789][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-stack]
[2022-07-21T16:27:46,789][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-text-structure]
[2022-07-21T16:27:46,790][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-voting-only-node]
[2022-07-21T16:27:46,790][INFO ][o.e.p.PluginsService     ] [es01] loaded module [x-pack-watcher]
[2022-07-21T16:27:46,790][INFO ][o.e.p.PluginsService     ] [es01] loaded plugin [mapper-murmur3]
[2022-07-21T16:27:46,791][INFO ][o.e.p.PluginsService     ] [es01] loaded plugin [mapper-size]
[2022-07-21T16:27:51,044][INFO ][o.e.e.NodeEnvironment    ] [es01] using [1] data paths, mounts [[/usr/share/elasticsearch/data_01 (/dev/mapper/centos-root)]], net usable_space [23.7gb], net total_space [47.7gb], types [xfs]
[2022-07-21T16:27:51,045][INFO ][o.e.e.NodeEnvironment    ] [es01] heap size [1gb], compressed ordinary object pointers [true]
[2022-07-21T16:27:51,135][INFO ][o.e.n.Node               ] [es01] node name [es01], node ID [9Q8W5kZ2RYGXwuw1MVteGQ], cluster name [dc02_es_cluster], roles [ml, data_hot, transform, data_content, data_warm, master, remote_cluster_client, data, data_cold, ingest, data_frozen]
[2022-07-21T16:27:55,902][INFO ][o.e.x.s.Security         ] [es01] Security is enabled
[2022-07-21T16:27:56,565][INFO ][o.e.x.s.a.s.FileRolesStore] [es01] parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]
[2022-07-21T16:27:57,185][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [es01] [controller/104] [Main.cc@123] controller (64 bit): Version 8.3.2 (Build c86b7174f20c42) Copyright (c) 2022 Elasticsearch BV
[2022-07-21T16:27:57,873][INFO ][o.e.t.n.NettyAllocator   ] [es01] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=1mb, factors={es.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=4mb, heap_size=1gb}]
[2022-07-21T16:27:57,912][INFO ][o.e.i.r.RecoverySettings ] [es01] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]
[2022-07-21T16:27:57,962][INFO ][o.e.d.DiscoveryModule    ] [es01] using discovery type [multi-node] and seed hosts providers [settings]

ylasri · August 3, 2022, 11:38am

Log messages go to the console and are handled by the configured Docker logging driver. By default you can access logs with docker logs. If you would prefer the Elasticsearch container to write logs to disk, set the ES_LOG_STYLE environment variable to file. This causes Elasticsearch to use the same logging configuration as other Elasticsearch distribution formats.

ylasri · August 3, 2022, 11:38am

More details here

github.com/elastic/elasticsearch

Custom log4j2.propoerties for Elasticsearch in docker

opened 09:30AM - 28 Jul 22 UTC

ylasri

>bug :Delivery/Packaging team-discuss Team:Delivery

### Elasticsearch Version 8.3.2 ### Installed Plugins N/A ### Java Version rel="noopener" class="show-more">…_bundled_ are handled by the configured Docker logging driver. To access logs we need to run docker logs $container_name. Elasticsearch to save logs into a specific path (same way if installed using RPM package for example) log4j2.properties with the same config that come from the RPM or TAR archive, but still not able to get logs into files in the specific path. tificates:ro' t:rw' custom log4j2.properties existing outputs are ignored. ] [es01] version[8.3.2], pid[70], build[docker/8b0b1f23fbebecc3c88e4464319dea8989f374fd/2022-07-06T15:15:15.901688194Z], OS[Linux/3.10.0-1160.71.1.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/18.0.1.1/18.0.1.1+2-6] ] [es01] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true] ] [es01] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -Des.cgroups.hierarchy.override=/, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-2879924629542642311, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms1g, -Xmx1g, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-2879924629542642311, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.distribution.type=docker, --module-path=/usr/share/elasticsearch/lib, -Djdk.module.main=org.elasticsearch.server] ] [es01] Package versions: jackson-annotations=2.13.2, jackson-core=2.13.2, jackson-databind=2.13.2.2, jackson-dataformat-xml=2.13.2, jackson-datatype-jsr310=2.13.2, azure-core=1.27.0, Troubleshooting version conflicts: https://aka.ms/azsdk/java/dependency/troubleshoot ] [es01] loaded module [aggs-matrix-stats] ] [es01] loaded module [analysis-common] ] [es01] loaded module [constant-keyword] ] [es01] loaded module [data-streams] ] [es01] loaded module [frozen-indices] ] [es01] loaded module [ingest-common] ] [es01] loaded module [ingest-geoip] ] [es01] loaded module [ingest-user-agent] ] [es01] loaded module [kibana] ] [es01] loaded module [lang-expression] ] [es01] loaded module [lang-mustache] ] [es01] loaded module [lang-painless] ] [es01] loaded module [legacy-geo] ] [es01] loaded module [mapper-extras] ] [es01] loaded module [mapper-version] ] [es01] loaded module [old-lucene-versions] ] [es01] loaded module [parent-join] ] [es01] loaded module [percolator] ] [es01] loaded module [rank-eval] ] [es01] loaded module [reindex] ] [es01] loaded module [repositories-metering-api] ] [es01] loaded module [repository-azure] ] [es01] loaded module [repository-encrypted] ] [es01] loaded module [repository-gcs] ] [es01] loaded module [repository-s3] ] [es01] loaded module [repository-url] ] [es01] loaded module [runtime-fields-common] ] [es01] loaded module [search-business-rules] ] [es01] loaded module [searchable-snapshots] ] [es01] loaded module [snapshot-based-recoveries] ] [es01] loaded module [snapshot-repo-test-kit] ] [es01] loaded module [spatial] ] [es01] loaded module [transform] ] [es01] loaded module [transport-netty4] ] [es01] loaded module [unsigned-long] ] [es01] loaded module [vector-tile] ] [es01] loaded module [vectors] ] [es01] loaded module [wildcard] ] [es01] loaded module [x-pack-aggregate-metric] ] [es01] loaded module [x-pack-analytics] ] [es01] loaded module [x-pack-async] ] [es01] loaded module [x-pack-async-search] ] [es01] loaded module [x-pack-autoscaling] ] [es01] loaded module [x-pack-ccr] ] [es01] loaded module [x-pack-core] ] [es01] loaded module [x-pack-deprecation] ] [es01] loaded module [x-pack-enrich] ] [es01] loaded module [x-pack-eql] ] [es01] loaded module [x-pack-fleet] ] [es01] loaded module [x-pack-graph] ] [es01] loaded module [x-pack-identity-provider] ] [es01] loaded module [x-pack-ilm] ] [es01] loaded module [x-pack-logstash] ] [es01] loaded module [x-pack-ml] ] [es01] loaded module [x-pack-monitoring] ] [es01] loaded module [x-pack-ql] ] [es01] loaded module [x-pack-rollup] ] [es01] loaded module [x-pack-security] ] [es01] loaded module [x-pack-shutdown] ] [es01] loaded module [x-pack-sql] ] [es01] loaded module [x-pack-stack] ] [es01] loaded module [x-pack-text-structure] ] [es01] loaded module [x-pack-voting-only-node] ] [es01] loaded module [x-pack-watcher] ] [es01] loaded plugin [mapper-murmur3] ] [es01] loaded plugin [mapper-size] ] [es01] using [1] data paths, mounts [[/usr/share/elasticsearch/data_01 (/dev/mapper/centos-root)]], net usable_space [23.7gb], net total_space [47.7gb], types [xfs] ] [es01] heap size [1gb], compressed ordinary object pointers [true] ] [es01] node name [es01], node ID [9Q8W5kZ2RYGXwuw1MVteGQ], cluster name [dc02_es_cluster], roles [ml, data_hot, transform, data_content, data_warm, master, remote_cluster_client, data, data_cold, ingest, data_frozen] ] [es01] Security is enabled [es01] parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml] [es01] [controller/104] [Main.cc@123] controller (64 bit): Version 8.3.2 (Build c86b7174f20c42) Copyright (c) 2022 Elasticsearch BV ] [es01] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=1mb, factors={es.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=4mb, heap_size=1gb}] ] [es01] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b] ] [es01] using discovery type [multi-node] and seed hosts providers [settings] example the same as used for Linux archive so that logs can be redirected directly to a folder)

system · August 31, 2022, 11:39am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Logstash 8.9.0 docker logs to stdout. how to provide custom path for it? Logstash docker	5	448	October 10, 2023
Where does Elasticsearch store/read logs Elasticsearch docker	2	1480	February 10, 2023
I can't find logs on elasticsearch Elasticsearch	9	312	September 28, 2022
Capturing Elasticsearch Logs Elasticsearch docker	3	419	January 15, 2019
ElasticSearch location in a docker Elasticsearch docker	2	275	September 4, 2022

Custom log4j2.propoerties for Elasticsearch in docker

Related Topics