Hello,
I'm new at Elasticsearch (ES) and i'm currently set a customized podman container ES 8.5.0 installation from base RPM repository
In this installation I'm using a dedicated Linux user 'elasticadm' which owns the files into the container and over the local Red Hat Linux 8.5 host
Basically I use the following ownership for the installation on localhost:
/app/elasticsearch/data - /var/log/elasticsearch/elasticsearch.log - /etc/elasticsearch/elasticsearch.yml:
elasticadm: elasticsearch - then after the below error occurred I tried: elasticadm:root (but with no more success)
I'm able to deploy the container without any bind-mount volumes as:
podman run --detach --name es850 --publish 9200:9200 --user=elasticadm localhost/elasticsearch_cust:1.4
podman logs es850
warning: ignoring JAVA_HOME=/usr/lib/jvm/java-openjdk; using bundled JDK
[2022-11-09T20:37:41,777][INFO ][o.e.n.Node ] [Prod] version[8.5.0], pid[72], build[rpm/c94b4700cda13820dad5aa74fae6db185ca5c304/2022-10-24T16:54:16.433628434Z], OS[Linux/4.18.0-348.7.1.el8_5.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/19/19+36-2238]
[2022-11-09T20:37:41,782][INFO ][o.e.n.Node ] [Prod] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2022-11-09T20:37:41,783][INFO ][o.e.n.Node ] [Prod] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-5358173424819503746, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms1868m, -Xmx1868m, -XX:MaxDirectMemorySize=979369984, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.distribution.type=rpm, --module-path=/usr/share/elasticsearch/lib, --add-modules=jdk.net, -Djdk.module.main=org.elasticsearch.server]
[2022-11-09T20:37:43,721][INFO ][c.a.c.i.j.JacksonVersion ] [Prod] Package versions: jackson-annotations=2.13.2, jackson-core=2.13.2, jackson-databind=2.13.2.2, jackson-dataformat-xml=2.13.2, jackson-datatype-jsr310=2.13.2, azure-core=1.27.0, Troubleshooting version conflicts: https://aka.ms/azsdk/java/dependency/troubleshoot
[2022-11-09T20:37:45,175][INFO ][o.e.p.PluginsService ] [Prod] loaded module [aggs-matrix-stats]
[2022-11-09T20:37:45,175][INFO ][o.e.p.PluginsService ] [Prod] loaded module [analysis-common]
[2022-11-09T20:37:45,176][INFO ][o.e.p.PluginsService ] [Prod] loaded module [apm]
......
[2022-11-09T20:37:45,190][INFO ][o.e.p.PluginsService ] [Prod] loaded module [x-pack-watcher]
[2022-11-09T20:37:45,191][INFO ][o.e.p.PluginsService ] [Prod] no plugins loaded
[2022-11-09T20:37:48,027][WARN ][stderr ] [Prod] Nov 09, 2022 8:37:48 PM org.apache.lucene.store.MMapDirectory lookupProvider
[2022-11-09T20:37:48,028][WARN ][stderr ] [Prod] WARNING: You are running with Java 19. To make full use of MMapDirectory, please pass '--enable-preview' to the Java command line.
[2022-11-09T20:37:48,039][INFO ][o.e.e.NodeEnvironment ] [Prod] using [1] data paths, mounts [[/ (overlay)]], net usable_space [24gb], net total_space [27.8gb], types [overlay]
[2022-11-09T20:37:48,039][INFO ][o.e.e.NodeEnvironment ] [Prod] heap size [1.8gb], compressed ordinary object pointers [true]
[2022-11-09T20:37:48,048][INFO ][o.e.n.Node ] [Prod] node name [Prod], node ID [CvroQFRsTxKqyWfwcOJGag], cluster name [elasticsearch], roles [data_frozen, ml, data_hot, transform, data_content, data_warm, master, remote_cluster_client, data, data_cold, ingest]
[2022-11-09T20:37:51,831][INFO ][o.e.x.s.Security ] [Prod] Security is enabled
[2022-11-09T20:37:52,214][INFO ][o.e.x.s.a.s.FileRolesStore] [Prod] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2022-11-09T20:37:52,628][INFO ][o.e.x.s.InitialNodeSecurityAutoConfiguration] [Prod] Auto-configuration will not generate a password for the elastic built-in superuser, as we cannot determine if there is a terminal attached to the elasticsearch process. You can use the `bin/elasticsearch-reset-password` tool to set the password for the elastic user.
[2022-11-09T20:37:52,724][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [Prod] [controller/96] [Main.cc@123] controller (64 bit): Version 8.5.0 (Build 3922fab346e761) Copyright (c) 2022 Elasticsearch BV
[2022-11-09T20:37:53,354][INFO ][o.e.t.n.NettyAllocator ] [Prod] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2022-11-09T20:37:53,381][INFO ][o.e.i.r.RecoverySettings ] [Prod] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]
[2022-11-09T20:37:53,425][INFO ][o.e.d.DiscoveryModule ] [Prod] using discovery type [single-node] and seed hosts providers [settings]
[2022-11-09T20:37:54,888][INFO ][o.e.n.Node ] [Prod] initialized
[2022-11-09T20:37:54,889][INFO ][o.e.n.Node ] [Prod] starting ...
[2022-11-09T20:37:54,901][INFO ][o.e.x.s.c.f.PersistentCache] [Prod] persistent cache index loaded
[2022-11-09T20:37:54,903][INFO ][o.e.x.d.l.DeprecationIndexingComponent] [Prod] deprecation component started
[2022-11-09T20:37:55,011][INFO ][o.e.t.TransportService ] [Prod] publish_address {10.0.2.100:9300}, bound_addresses {[::]:9300}
[2022-11-09T20:37:55,122][WARN ][o.e.b.BootstrapChecks ] [Prod] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2022-11-09T20:37:55,124][INFO ][o.e.c.c.ClusterBootstrapService] [Prod] this node has not joined a bootstrapped cluster yet; [cluster.initial_master_nodes] is set to [Prod]
[2022-11-09T20:37:55,133][INFO ][o.e.c.c.Coordinator ] [Prod] setting initial configuration to VotingConfiguration{CvroQFRsTxKqyWfwcOJGag}
[2022-11-09T20:37:55,327][INFO ][o.e.c.s.MasterService ] [Prod] elected-as-master ([1] nodes joined)[_FINISH_ELECTION_, {Prod}{CvroQFRsTxKqyWfwcOJGag}{oYVn8g0ZS2CFxHKYosdd_Q}{Prod}{10.0.2.100}{10.0.2.100:9300}{cdfhilmrstw} completing election], term: 1, version: 1, delta: master node changed {previous [], current [{Prod}{CvroQFRsTxKqyWfwcOJGag}{oYVn8g0ZS2CFxHKYosdd_Q}{Prod}{10.0.2.100}{10.0.2.100:9300}{cdfhilmrstw}]}
[2022-11-09T20:37:55,352][INFO ][o.e.c.c.CoordinationState] [Prod] cluster UUID set to [_wcBh4-JRtuLqIBXyNhZ5A]
[2022-11-09T20:37:55,370][INFO ][o.e.c.s.ClusterApplierService] [Prod] master node changed {previous [], current [{Prod}{CvroQFRsTxKqyWfwcOJGag}{oYVn8g0ZS2CFxHKYosdd_Q}{Prod}{10.0.2.100}{10.0.2.100:9300}{cdfhilmrstw}]}, term: 1, version: 1, reason: Publication{term=1, version=1}
[2022-11-09T20:37:55,439][INFO ][o.e.r.s.FileSettingsService] [Prod] starting file settings watcher ...
[2022-11-09T20:37:55,447][INFO ][o.e.r.s.FileSettingsService] [Prod] file settings service up and running [tid=51]
[2022-11-09T20:37:55,456][INFO ][o.e.h.AbstractHttpServerTransport] [Prod] publish_address {10.0.2.100:9200}, bound_addresses {[::]:9200}
[2022-11-09T20:37:55,457][INFO ][o.e.n.Node ] [Prod] started {Prod}{CvroQFRsTxKqyWfwcOJGag}{oYVn8g0ZS2CFxHKYosdd_Q}{Prod}{10.0.2.100}{10.0.2.100:9300}{cdfhilmrstw}{ml.max_jvm_size=1958739968, ml.allocated_processors_double=4.0, xpack.installed=true, ml.machine_memory=3917570048, ml.allocated_processors=4}
[2022-11-09T20:37:55,510][INFO ][o.e.g.GatewayService ] [Prod] recovered [0] indices into cluster_state
[2022-11-09T20:37:55,691][INFO ][o.e.c.m.MetadataIndexTemplateService] [Prod] adding index template [.watch-history-16] for index patterns [.watcher-history-16*]
[2022-11-09T20:37:55,700][INFO ][o.e.c.m.MetadataIndexTemplateService] [Prod] adding index template [ilm-history] for index patterns [ilm-history-5*]
[2022-11-09T20:37:55,707][INFO ][o.e.c.m.MetadataIndexTemplateService] [Prod] adding index template [.slm-history] for index patterns [.slm-history-5*]
[2022-11-09T20:37:55,718][INFO ][o.e.c.m.MetadataIndexTemplateService] [Prod] adding component template [.deprecation-indexing-mappings]
[2022-11-09T20:37:55,723][INFO ][o.e.c.m.MetadataIndexTemplateService] [Prod] adding component template [synthetics-mappings]
......
[2022-11-09T20:37:56,392][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [Prod] adding index lifecycle policy [.fleet-actions-results-ilm-policy]
[2022-11-09T20:37:56,510][INFO ][o.e.l.LicenseService ] [Prod] license [4b5d6876-1402-470e-96fd-f9ff8211cca7] mode [basic] - valid
[2022-11-09T20:37:56,511][INFO ][o.e.x.s.a.Realms ] [Prod] license mode is [basic], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]
[2022-11-09T20:37:56,538][INFO ][o.e.h.n.s.HealthNodeTaskExecutor] [Prod] Node [{Prod}{CvroQFRsTxKqyWfwcOJGag}] is selected as the current health node.
# Configuration is the following (elasticsearch.yml):
node.name: Prod # Name is 'Prod' but it's not a true production server
path.data: /app/elasticsearch/data
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
discovery.type: single-node
ingest.geoip.downloader.enabled: false
# Security:
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
enabled: true
keystore.path: certs/http.p12
# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
keystore.path: certs/transport.p12
truststore.path: certs/transport.p12
http.host: 0.0.0.0
#transport.host: 0.0.0.0
$ podman exec -it es850 bash
[elasticadm@8a9ceb50b3b4 /]$ /usr/share/elasticsearch/bin/elasticsearch-keystore list
warning: ignoring JAVA_HOME=/usr/lib/jvm/java-openjdk; using bundled JDK
autoconfiguration.password_hash
keystore.seed
xpack.security.http.ssl.keystore.secure_password
xpack.security.transport.ssl.keystore.secure_password
xpack.security.transport.ssl.truststore.secure_password
and connection test is fine:
curl --cacert http_ca.crt -u elastic https://127.0.0.1:9200
Enter host password for user 'elastic':
{
"name" : "Prod",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "........",
"version" : {
"number" : "8.5.0",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "c94b4700cda13820dad5aa74fae6db185ca5c304",
"build_date" : "2022-10-24T16:54:16.433628434Z",
"build_snapshot" : false,
"lucene_version" : "9.4.1",
"minimum_wire_compatibility_version" : "7.17.0",
"minimum_index_compatibility_version" : "7.0.0"
},
"tagline" : "You Know, for Search"
Whenever I execute the same command with any mount-bind volumes the installation fails with the following error message I could not explain - I had a close eye on the related security configuration -but I maybe missed some something:
**fatal exception while booting Elasticsearch org.elasticsearch.ElasticsearchSecurityException: invalid configuration for xpack.security.transport.ssl - [xpack.security.transport.ssl.enabled] is not set, but the following settings have been configured in elasticsearch.yml : [xpack.security.transport.ssl.keystore.secure_password,xpack.security.transport.ssl.truststore.secure_password]**
podman run --detach --name es850 --publish 9200:9200
--volume=/etc/elasticsearch/elasticsearch.yml:/etc/elasticsearch/elasticsearch.yml :Z
--volume=/var/log/elasticsearch/elasticsearch.log:/var/log/elasticsearch/elasticsearch.log:Z
--volume=/app/elasticsearch/data:/app/elasticsearch/data:Z
--user=elasticadm localhost/elasticsearch_cust:1.4
podman logs es850
warning: ignoring JAVA_HOME=/usr/lib/jvm/java-openjdk; using bundled JDK
Aborting auto configuration because the node keystore contains password settings already
[2022-11-09T15:56:27,292][INFO ][o.e.n.Node ] [0d8414e9b51b] version[8.5.0], pid[76], build[rpm/c94b4700cda13820dad5aa74fae6db185ca5c304/2022-10-24T16:54:16.433628434Z], OS[Linux/4.18.0-348.7.1.el8_5.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/19/19+36-2238]
[2022-11-09T15:56:27,299][INFO ][o.e.n.Node ] [0d8414e9b51b] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2022-11-09T15:56:27,300][INFO ][o.e.n.Node ] [0d8414e9b51b] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-10492222574682252504, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms1868m, -Xmx1868m, -XX:MaxDirectMemorySize=979369984, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.distribution.type=rpm, --module-path=/usr/share/elasticsearch/lib, --add-modules=jdk.net, -Djdk.module.main=org.elasticsearch.server]
[2022-11-09T15:56:29,369][INFO ][c.a.c.i.j.JacksonVersion ] [0d8414e9b51b] Package versions: jackson-annotations=2.13.2, jackson-core=2.13.2, jackson-databind=2.13.2.2, jackson-dataformat-xml=2.13.2, jackson-datatype-jsr310=2.13.2, azure-core=1.27.0, Troubleshooting version conflicts: https://aka.ms/azsdk/java/dependency/troubleshoot
[2022-11-09T15:56:30,863][INFO ][o.e.p.PluginsService ] [0d8414e9b51b] loaded module [aggs-matrix-stats]
.............
[2022-11-09T15:56:30,880][INFO ][o.e.p.PluginsService ] [0d8414e9b51b] loaded module [x-pack-watcher]
[2022-11-09T15:56:30,881][INFO ][o.e.p.PluginsService ] [0d8414e9b51b] no plugins loaded
[2022-11-09T15:56:33,720][WARN ][stderr ] [0d8414e9b51b] Nov 09, 2022 3:56:33 PM org.apache.lucene.store.MMapDirectory lookupProvider
[2022-11-09T15:56:33,721][WARN ][stderr ] [0d8414e9b51b] WARNING: You are running with Java 19. To make full use of MMapDirectory, please pass '--enable-preview' to the Java command line.
[2022-11-09T15:56:33,732][INFO ][o.e.e.NodeEnvironment ] [0d8414e9b51b] using [1] data paths, mounts [[/ (overlay)]], net usable_space [24gb], net total_space [27.8gb], types [overlay]
[2022-11-09T15:56:33,732][INFO ][o.e.e.NodeEnvironment ] [0d8414e9b51b] heap size [1.8gb], compressed ordinary object pointers [true]
[2022-11-09T15:56:33,740][INFO ][o.e.n.Node ] [0d8414e9b51b] node name [0d8414e9b51b], node ID [rMFgxntETo63opwgU7P9sg], cluster name [elasticsearch], roles [ml, data_hot, transform, data_content, data_warm, master, remote_cluster_client, data, data_cold, ingest, data_frozen]
[2022-11-09T15:56:36,194][ERROR][o.e.b.Elasticsearch **] [0d8414e9b51b] fatal exception while booting Elasticsearch org.elasticsearch.ElasticsearchSecurityException: invalid configuration for xpack.security.transport.ssl - [xpack.security.transport.ssl.enabled] is not set, but the following settings have been configured in elasticsearch.yml : [xpack.security.transport.ssl.keystore.secure_password,xpack.security.transport.ssl.truststore.secure_password]**
at org.elasticsearch.xcore@8.5.0/org.elasticsearch.xpack.core.ssl.SSLService.validateServerConfiguration(SSLService.java:648)
at org.elasticsearch.xcore@8.5.0/org.elasticsearch.xpack.core.ssl.SSLService.loadSslConfigurations(SSLService.java:612)
at org.elasticsearch.xcore@8.5.0/org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:156)
at org.elasticsearch.xcore@8.5.0/org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:465)
at org.elasticsearch.xcore@8.5.0/org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:314)
at org.elasticsearch.server@8.5.0/org.elasticsearch.node.Node.lambda$new$15(Node.java:704)
at org.elasticsearch.server@8.5.0/org.elasticsearch.plugins.PluginsService.lambda$flatMap$0(PluginsService.java:252)
at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:722)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575)
at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616)
at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622)
at java.base/java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627)
at org.elasticsearch.server@8.5.0/org.elasticsearch.node.Node.<init>(Node.java:719)
at org.elasticsearch.server@8.5.0/org.elasticsearch.node.Node.<init>(Node.java:316)
at org.elasticsearch.server@8.5.0/org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:214)
at org.elasticsearch.server@8.5.0/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:214)
at org.elasticsearch.server@8.5.0/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67)
ERROR: Elasticsearch did not exit normally - check the logs at /usr/share/elasticsearch/logs/elasticsearch.log
Any ideas / advise would be really appreciated because I don't know what's wrong suddenly with xpack.security parameters and the relationship with the podman bind-mount volume ?
These base xpack.security seem well configured (initial base configuration with no modification in a first time)
Best regards
Dominique