When I add xpack to stack it is not working

When trying to implement xpack on my stack I get an "error loading data" message in Cerebro and it does not let me login. when set to false all works - when set to true it does not.

here is info on the stack

{
  "name" : "prod-resultselasticsearchm-101w",
  "cluster_name" : "prod-resultselasticsearch",
  "cluster_uuid" : "ztoDRCgNTome-NHt6sZk1g",
  "version" : {
    "number" : "7.17.5",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "8d61b4f7ddf931f219e3745f295ed2bbc50c8e84",
    "build_date" : "2022-06-23T21:57:28.736740635Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

this next part is elasticsearch.yml on hosts - I checked the node.data settings on each host and they are correct

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: prod-resultselasticsearch
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: prod-resultselasticsearchm-101w
node.data: false
node.master: true
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /elasticsearch/data
#
# Path to log files:
#
path.logs: /elasticsearch/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
bootstrap.system_call_filter: false
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 10.212.39.91
#
# Set a custom port for HTTP:
#
http.port: 9200
#
transport.tcp.port: 9300
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["10.212.39.91", "10.212.39.92", "10.212.39.93", "10.212.39.94", "10.212.39.95"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["10.212.39.91"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
gateway.recover_after_nodes: 5
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
#
############################Custom########################################
cluster.remote.initial_connect_timeout: 30s
cluster.routing.allocation.node_concurrent_recoveries: 2
cluster.routing.allocation.node_initial_primaries_recoveries: 3
http.cors.enabled: true
indices.fielddata.cache.size: 20%
indices.queries.cache.size: 20%
indices.recovery.max_bytes_per_sec: 15mb
# For truble shooting elastic search issues #
#logger.org.elasticsearch.cluster.coordination.ClusterBootstrapService: TRACE
#logger.org.elasticsearch.discovery: TRACE
############################################################################
#BEGIN Configure Baisc Authentication
xpack.security.enabled: false
xpack.security.transport.ssl.enabled: false
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
#END Configure Baisc Authentication
#BEGIN Enable SSL
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.verification_mode: certificate
xpack.security.http.ssl.keystore.path: elastic-certificates.p12
xpack.security.http.ssl.truststore.path: elastic-certificates.p12
#END Enable SSL
#reindex.remote.whitelist: "10.212.39.52:9200"

all works fine and dandy.

when I set -

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true

it does not. I can give you more info if you need

Sharing your Elasticsearch logs would be helpful.

below is an log before a restart - then after a restart

[root@prod-resultselasticsearchm-101w logs]# cat prod-resultselasticsearch.log

[2022-09-16T20:15:28,242][INFO ][o.e.i.g.DatabaseNodeService] [prod-resultselasticsearchm-101w] retrieve geoip database [GeoLite2-ASN.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-3165155470998174077/geoip-databases/ZN56Y0twTkaQOVlD86bPDQ/GeoLite2-ASN.mmdb.tmp.gz]
[2022-09-16T20:15:28,760][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [prod-resultselasticsearchm-101w] evicted [0] entries from cache after reloading database [/tmp/elasticsearch-3165155470998174077/geoip-databases/ZN56Y0twTkaQOVlD86bPDQ/GeoLite2-ASN.mmdb]
[2022-09-16T20:15:28,760][INFO ][o.e.i.g.DatabaseNodeService] [prod-resultselasticsearchm-101w] successfully reloaded changed geoip database file [/tmp/elasticsearch-3165155470998174077/geoip-databases/ZN56Y0twTkaQOVlD86bPDQ/GeoLite2-ASN.mmdb]
[2022-09-16T20:15:32,049][INFO ][o.e.i.g.DatabaseNodeService] [prod-resultselasticsearchm-101w] retrieve geoip database [GeoLite2-City.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-3165155470998174077/geoip-databases/ZN56Y0twTkaQOVlD86bPDQ/GeoLite2-City.mmdb.tmp.gz]
[2022-09-16T20:15:33,593][INFO ][o.e.i.g.DatabaseNodeService] [prod-resultselasticsearchm-101w] retrieve geoip database [GeoLite2-Country.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-3165155470998174077/geoip-databases/ZN56Y0twTkaQOVlD86bPDQ/GeoLite2-Country.mmdb.tmp.gz]
[2022-09-16T20:15:33,738][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [prod-resultselasticsearchm-101w] evicted [0] entries from cache after reloading database [/tmp/elasticsearch-3165155470998174077/geoip-databases/ZN56Y0twTkaQOVlD86bPDQ/GeoLite2-Country.mmdb]
[2022-09-16T20:15:33,738][INFO ][o.e.i.g.DatabaseNodeService] [prod-resultselasticsearchm-101w] successfully reloaded changed geoip database file [/tmp/elasticsearch-3165155470998174077/geoip-databases/ZN56Y0twTkaQOVlD86bPDQ/GeoLite2-Country.mmdb]
[2022-09-16T20:15:33,998][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [prod-resultselasticsearchm-101w] evicted [0] entries from cache after reloading database [/tmp/elasticsearch-3165155470998174077/geoip-databases/ZN56Y0twTkaQOVlD86bPDQ/GeoLite2-City.mmdb]
[2022-09-16T20:15:33,999][INFO ][o.e.i.g.DatabaseNodeService] [prod-resultselasticsearchm-101w] successfully reloaded changed geoip database file [/tmp/elasticsearch-3165155470998174077/geoip-databases/ZN56Y0twTkaQOVlD86bPDQ/GeoLite2-City.mmdb]

[root@prod-resultselasticsearchm-101w logs]# systemctl restart elasticsearch
[root@prod-resultselasticsearchm-101w logs]# cat prod-resultselasticsearch.log

[2022-09-19T13:10:47,433][INFO ][o.e.n.Node               ] [prod-resultselasticsearchm-101w] stopping ...
[2022-09-19T13:10:47,449][INFO ][o.e.x.w.WatcherService   ] [prod-resultselasticsearchm-101w] stopping watch service, reason [shutdown initiated]
[2022-09-19T13:10:47,450][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [prod-resultselasticsearchm-101w] [controller/61242] [Main.cc@174] ML controller exiting
[2022-09-19T13:10:47,451][INFO ][o.e.x.m.p.NativeController] [prod-resultselasticsearchm-101w] Native controller process has stopped - no new native processes can be started
[2022-09-19T13:10:47,452][INFO ][o.e.x.w.WatcherLifeCycleService] [prod-resultselasticsearchm-101w] watcher has stopped and shutdown
[2022-09-19T13:10:47,453][INFO ][o.e.c.c.Coordinator      ] [prod-resultselasticsearchm-101w] master node [{prod-resultselasticsearchm-103w}{Q0xgQdbEQJGEAEFKM6On3w}{rXBiNTcJQ7qQo7mRPc3cjQ}{10.212.39.93}{10.212.39.93:9300}{ilmr}] disconnected, restarting discovery
[2022-09-19T13:10:47,473][INFO ][o.e.n.Node               ] [prod-resultselasticsearchm-101w] stopped
[2022-09-19T13:10:47,473][INFO ][o.e.n.Node               ] [prod-resultselasticsearchm-101w] closing ...
[2022-09-19T13:10:47,483][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [prod-resultselasticsearchm-101w] evicted [0] entries from cache after reloading database [/tmp/elasticsearch-3165155470998174077/geoip-databases/ZN56Y0twTkaQOVlD86bPDQ/GeoLite2-Country.mmdb]
[2022-09-19T13:10:47,483][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [prod-resultselasticsearchm-101w] evicted [0] entries from cache after reloading database [/tmp/elasticsearch-3165155470998174077/geoip-databases/ZN56Y0twTkaQOVlD86bPDQ/GeoLite2-ASN.mmdb]
[2022-09-19T13:10:47,483][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [prod-resultselasticsearchm-101w] evicted [0] entries from cache after reloading database [/tmp/elasticsearch-3165155470998174077/geoip-databases/ZN56Y0twTkaQOVlD86bPDQ/GeoLite2-City.mmdb]
[2022-09-19T13:10:47,484][INFO ][o.e.n.Node               ] [prod-resultselasticsearchm-101w] closed
[2022-09-19T13:10:50,272][INFO ][o.e.n.Node               ] [prod-resultselasticsearchm-101w] version[7.17.5], pid[106803], build[default/rpm/8d61b4f7ddf931f219e3745f295ed2bbc50c8e84/2022-06-23T21:57:28.736740635Z], OS[Linux/3.10.0-1160.71.1.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/18.0.1.1/18.0.1.1+2-6]
[2022-09-19T13:10:50,273][INFO ][o.e.n.Node               ] [prod-resultselasticsearchm-101w] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2022-09-19T13:10:50,273][INFO ][o.e.n.Node               ] [prod-resultselasticsearchm-101w] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -Djava.security.manager=allow, -Xms4g, -Xmx4g, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-4947918882176097480, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=/elasticsearch/logs, -XX:ErrorFile=/elasticsearch/logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/elasticsearch/logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=2147483648, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
[2022-09-19T13:10:52,167][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [aggs-matrix-stats]
[2022-09-19T13:10:52,167][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [analysis-common]
[2022-09-19T13:10:52,167][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [constant-keyword]
[2022-09-19T13:10:52,168][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [frozen-indices]
[2022-09-19T13:10:52,168][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [ingest-common]
[2022-09-19T13:10:52,168][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [ingest-geoip]
[2022-09-19T13:10:52,168][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [ingest-user-agent]
[2022-09-19T13:10:52,168][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [kibana]
[2022-09-19T13:10:52,168][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [lang-expression]
[2022-09-19T13:10:52,169][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [lang-mustache]
[2022-09-19T13:10:52,169][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [lang-painless]
[2022-09-19T13:10:52,169][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [legacy-geo]
[2022-09-19T13:10:52,169][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [mapper-extras]
[2022-09-19T13:10:52,169][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [mapper-version]
[2022-09-19T13:10:52,169][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [parent-join]
[2022-09-19T13:10:52,169][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [percolator]
[2022-09-19T13:10:52,170][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [rank-eval]
[2022-09-19T13:10:52,170][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [reindex]
[2022-09-19T13:10:52,170][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [repositories-metering-api]
[2022-09-19T13:10:52,170][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [repository-encrypted]
[2022-09-19T13:10:52,170][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [repository-url]
[2022-09-19T13:10:52,170][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [runtime-fields-common]
[2022-09-19T13:10:52,170][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [search-business-rules]
[2022-09-19T13:10:52,171][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [searchable-snapshots]
[2022-09-19T13:10:52,171][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [snapshot-repo-test-kit]
[2022-09-19T13:10:52,171][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [spatial]
[2022-09-19T13:10:52,171][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [systemd]
[2022-09-19T13:10:52,171][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [transform]
[2022-09-19T13:10:52,171][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [transport-netty4]
[2022-09-19T13:10:52,171][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [unsigned-long]
[2022-09-19T13:10:52,172][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [vector-tile]
[2022-09-19T13:10:52,172][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [vectors]
[2022-09-19T13:10:52,172][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [wildcard]
[2022-09-19T13:10:52,172][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [x-pack-aggregate-metric]
[2022-09-19T13:10:52,172][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [x-pack-analytics]
[2022-09-19T13:10:52,172][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [x-pack-async]
[2022-09-19T13:10:52,172][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [x-pack-async-search]
[2022-09-19T13:10:52,173][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [x-pack-autoscaling]
[2022-09-19T13:10:52,173][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [x-pack-ccr]
[2022-09-19T13:10:52,173][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [x-pack-core]
[2022-09-19T13:10:52,173][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [x-pack-data-streams]
[2022-09-19T13:10:52,173][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [x-pack-deprecation]
[2022-09-19T13:10:52,173][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [x-pack-enrich]
[2022-09-19T13:10:52,173][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [x-pack-eql]
[2022-09-19T13:10:52,174][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [x-pack-fleet]
[2022-09-19T13:10:52,174][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [x-pack-graph]
[2022-09-19T13:10:52,174][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [x-pack-identity-provider]
[2022-09-19T13:10:52,174][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [x-pack-ilm]
[2022-09-19T13:10:52,174][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [x-pack-logstash]
[2022-09-19T13:10:52,174][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [x-pack-ml]
[2022-09-19T13:10:52,174][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [x-pack-monitoring]
[2022-09-19T13:10:52,175][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [x-pack-ql]
[2022-09-19T13:10:52,175][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [x-pack-rollup]
[2022-09-19T13:10:52,175][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [x-pack-security]
[2022-09-19T13:10:52,175][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [x-pack-shutdown]
[2022-09-19T13:10:52,175][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [x-pack-sql]
[2022-09-19T13:10:52,175][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [x-pack-stack]
[2022-09-19T13:10:52,175][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [x-pack-text-structure]
[2022-09-19T13:10:52,175][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [x-pack-voting-only-node]
[2022-09-19T13:10:52,176][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] loaded module [x-pack-watcher]
[2022-09-19T13:10:52,176][INFO ][o.e.p.PluginsService     ] [prod-resultselasticsearchm-101w] no plugins loaded
[2022-09-19T13:10:52,210][INFO ][o.e.e.NodeEnvironment    ] [prod-resultselasticsearchm-101w] using [1] data paths, mounts [[/elasticsearch (/dev/mapper/secret)]], net usable_space [9.1gb], net total_space [9.7gb], types [ext4]
[2022-09-19T13:10:52,210][INFO ][o.e.e.NodeEnvironment    ] [prod-resultselasticsearchm-101w] heap size [4gb], compressed ordinary object pointers [true]
[2022-09-19T13:10:52,261][INFO ][o.e.n.Node               ] [prod-resultselasticsearchm-101w] node name [prod-resultselasticsearchm-101w], node ID [ZN56Y0twTkaQOVlD86bPDQ], cluster name [prod-resultselasticsearch], roles [master, remote_cluster_client, ml, ingest]
[2022-09-19T13:10:56,573][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [prod-resultselasticsearchm-101w] [controller/107002] [Main.cc@122] controller (64 bit): Version 7.17.5 (Build 6070d498c77671) Copyright (c) 2022 Elasticsearch BV
[2022-09-19T13:10:56,882][INFO ][o.e.i.g.ConfigDatabases  ] [prod-resultselasticsearchm-101w] initialized default databases [[GeoLite2-Country.mmdb, GeoLite2-City.mmdb, GeoLite2-ASN.mmdb]], config databases [[]] and watching [/etc/elasticsearch/ingest-geoip] for changes
[2022-09-19T13:10:56,883][INFO ][o.e.i.g.DatabaseNodeService] [prod-resultselasticsearchm-101w] initialized database registry, using geoip-databases directory [/tmp/elasticsearch-4947918882176097480/geoip-databases/ZN56Y0twTkaQOVlD86bPDQ]
[2022-09-19T13:10:57,363][INFO ][o.e.t.NettyAllocator     ] [prod-resultselasticsearchm-101w] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2022-09-19T13:10:57,386][INFO ][o.e.i.r.RecoverySettings ] [prod-resultselasticsearchm-101w] using rate limit [15mb] with [default=15mb, read=0b, write=0b, max=0b]
[2022-09-19T13:10:57,417][INFO ][o.e.d.DiscoveryModule    ] [prod-resultselasticsearchm-101w] using discovery type [zen] and seed hosts providers [settings]
[2022-09-19T13:10:57,797][INFO ][o.e.g.DanglingIndicesState] [prod-resultselasticsearchm-101w] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2022-09-19T13:10:58,351][INFO ][o.e.n.Node               ] [prod-resultselasticsearchm-101w] initialized
[2022-09-19T13:10:58,351][INFO ][o.e.n.Node               ] [prod-resultselasticsearchm-101w] starting ...
[2022-09-19T13:10:58,354][INFO ][o.e.x.d.l.DeprecationIndexingComponent] [prod-resultselasticsearchm-101w] deprecation component started
[2022-09-19T13:10:58,433][INFO ][o.e.t.TransportService   ] [prod-resultselasticsearchm-101w] publish_address {10.212.39.91:9300}, bound_addresses {10.212.39.91:9300}
[2022-09-19T13:10:58,711][INFO ][o.e.b.BootstrapChecks    ] [prod-resultselasticsearchm-101w] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2022-09-19T13:10:58,713][INFO ][o.e.c.c.Coordinator      ] [prod-resultselasticsearchm-101w] cluster UUID [ztoDRCgNTome-NHt6sZk1g]
[2022-09-19T13:10:58,939][INFO ][o.e.c.s.ClusterApplierService] [prod-resultselasticsearchm-101w] master node changed {previous [], current [{prod-resultselasticsearchm-103w}{Q0xgQdbEQJGEAEFKM6On3w}{rXBiNTcJQ7qQo7mRPc3cjQ}{10.212.39.93}{10.212.39.93:9300}{ilmr}]}, added {{prod-resultselasticsearchd-105w}{Iij5WOV8QIqHVcRL-snQPQ}{GsUqbklKTJOzTEaoLbdOIw}{10.212.39.100}{10.212.39.100:9300}{cdfhilrstw}, {prod-resultselasticsearchd-102w}{osq8RkuFSWyqaUTaejwVFQ}{znmszTK9TnCUjkNmn5YirQ}{10.212.39.97}{10.212.39.97:9300}{cdfhilrstw}, {prod-resultselasticsearchm-103w}{Q0xgQdbEQJGEAEFKM6On3w}{rXBiNTcJQ7qQo7mRPc3cjQ}{10.212.39.93}{10.212.39.93:9300}{ilmr}, {prod-resultselasticsearchd-101w}{3BDuEpdkRJKggzrWdS9zMA}{pllY9XMqRcOR24LIxwyVwg}{10.212.39.96}{10.212.39.96:9300}{cdfhilrstw}, {prod-resultselasticsearchm-102w}{LjJlMVMJSRWgeyKtyINfcg}{V5AFlcLdR0CjDh189zAwKA}{10.212.39.92}{10.212.39.92:9300}{ilmr}, {prod-resultselasticsearchm-105w}{pL2YO8XyRgq96fZ6YE3qgg}{KtLZs7PwR9KUnBOmmVOv8g}{10.212.39.95}{10.212.39.95:9300}{ilmr}, {prod-resultselasticsearchd-110w}{r5xpKmoiQQKe0cyS2ShHxw}{mNEdt9z7RCWzCzu6C7hPcA}{10.212.39.105}{10.212.39.105:9300}{cdfhilrstw}, {prod-resultselasticsearchd-109w}{i8WCq64mQ4-nrJX0r4AHGg}{XUL7eCYqR7C6hMd6pfUhHA}{10.212.39.104}{10.212.39.104:9300}{cdfhilrstw}, {prod-resultselasticsearchd-107w}{NnIiBZB0RZeIkEDDSYgjCw}{CcvKLdkJSY2bdHvzW1awqA}{10.212.39.102}{10.212.39.102:9300}{cdfhilrstw}, {prod-resultselasticsearchd-106w}{HDttVhUiSySauKb6rilrMA}{1MwzqAeYSw6iiplirXYYvQ}{10.212.39.101}{10.212.39.101:9300}{cdfhilrstw}, {prod-resultselasticsearchc-102w}{L8dAHMApQb-UnfgsOTdk3A}{nc_szeNJR1eum4tfWqpvQw}{10.212.39.107}{10.212.39.107:9300}{ilr}, {prod-resultselasticsearchc-103w}{qS8r9qsBTGCdztJD0GoQLQ}{21nq-DZKRZepjT350XQ75Q}{10.212.39.108}{10.212.39.108:9300}{ilr}, {prod-resultselasticsearchd-103w}{6kYumoEZTsuPukfNnscaqw}{-mNyesuKT9aAQ9htXzEiDA}{10.212.39.98}{10.212.39.98:9300}{cdfhilrstw}, {prod-resultselasticsearchd-108w}{YzwOTkz1RLeSNPBNZdb4kg}{y4zFYTBgT7CUMXEQlZWvsQ}{10.212.39.103}{10.212.39.103:9300}{cdfhilrstw}, {prod-resultselasticsearchd-104w}{z2sXRtOQQUOWoMS6rj18Nw}{8VNNjZz4RUiLaxC8VL6N7A}{10.212.39.99}{10.212.39.99:9300}{cdfhilrstw}, {prod-resultselasticsearchc-101w}{DE8r-HjwQmWxWy2gsD2F3A}{QqqzL_E1Sq2ij-tLI40PJw}{10.212.39.106}{10.212.39.106:9300}{ilr}, {prod-resultselasticsearchm-104w}{H1656CU4R5S8SbiKQcyxuA}{8MgjKpCbSsuDyWy3lN0OKw}{10.212.39.94}{10.212.39.94:9300}{ilmr}}, term: 80, version: 1338, reason: ApplyCommitRequest{term=80, version=1338, sourceNode={prod-resultselasticsearchm-103w}{Q0xgQdbEQJGEAEFKM6On3w}{rXBiNTcJQ7qQo7mRPc3cjQ}{10.212.39.93}{10.212.39.93:9300}{ilmr}{ml.machine_memory=8181776384, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=4294967296, transform.node=false}}
[2022-09-19T13:10:58,992][INFO ][o.e.i.g.DatabaseNodeService] [prod-resultselasticsearchm-101w] retrieve geoip database [GeoLite2-Country.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-4947918882176097480/geoip-databases/ZN56Y0twTkaQOVlD86bPDQ/GeoLite2-Country.mmdb.tmp.gz]
[2022-09-19T13:10:58,993][INFO ][o.e.i.g.DatabaseNodeService] [prod-resultselasticsearchm-101w] retrieve geoip database [GeoLite2-City.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-4947918882176097480/geoip-databases/ZN56Y0twTkaQOVlD86bPDQ/GeoLite2-City.mmdb.tmp.gz]
[2022-09-19T13:10:58,994][INFO ][o.e.i.g.DatabaseNodeService] [prod-resultselasticsearchm-101w] retrieve geoip database [GeoLite2-ASN.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-4947918882176097480/geoip-databases/ZN56Y0twTkaQOVlD86bPDQ/GeoLite2-ASN.mmdb.tmp.gz]
[2022-09-19T13:10:58,999][ERROR][o.e.i.g.DatabaseNodeService] [prod-resultselasticsearchm-101w] failed to retrieve database [GeoLite2-Country.mmdb]
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:179) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:165) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:929) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:763) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$6(TransportSearchAction.java:399) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:112) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:77) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:487) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:285) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:101) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:179) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:154) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:82) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:95) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:73) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:407) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:57) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.client.OriginSettingClient.doExecute(OriginSettingClient.java:51) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:407) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:392) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:542) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.ingest.geoip.DatabaseNodeService.lambda$retrieveDatabase$11(DatabaseNodeService.java:376) [ingest-geoip-7.17.5.jar:7.17.5]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.17.5.jar:7.17.5]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-09-19T13:10:59,006][ERROR][o.e.i.g.DatabaseNodeService] [prod-resultselasticsearchm-101w] failed to retrieve database [GeoLite2-City.mmdb]
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:179) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:165) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:929) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:763) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$6(TransportSearchAction.java:399) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:112) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:77) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:487) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:285) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:101) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:179) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:154) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:82) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:95) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:73) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:407) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:57) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.client.OriginSettingClient.doExecute(OriginSettingClient.java:51) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:407) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:392) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:542) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.ingest.geoip.DatabaseNodeService.lambda$retrieveDatabase$11(DatabaseNodeService.java:376) [ingest-geoip-7.17.5.jar:7.17.5]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.17.5.jar:7.17.5]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-09-19T13:10:59,007][ERROR][o.e.i.g.DatabaseNodeService] [prod-resultselasticsearchm-101w] failed to retrieve database [GeoLite2-ASN.mmdb]
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:179) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:165) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:929) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:763) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$6(TransportSearchAction.java:399) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:112) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:77) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:487) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:285) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:101) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:179) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:154) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:82) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:95) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:73) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:407) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:57) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.client.OriginSettingClient.doExecute(OriginSettingClient.java:51) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:407) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:392) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:542) ~[elasticsearch-7.17.5.jar:7.17.5]
        at org.elasticsearch.ingest.geoip.DatabaseNodeService.lambda$retrieveDatabase$11(DatabaseNodeService.java:376) [ingest-geoip-7.17.5.jar:7.17.5]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.17.5.jar:7.17.5]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-09-19T13:10:59,201][INFO ][o.e.l.LicenseService     ] [prod-resultselasticsearchm-101w] license [f069c374-fbe5-436c-ad8d-f8ebd6d61fe9] mode [basic] - valid
[2022-09-19T13:10:59,212][INFO ][o.e.h.AbstractHttpServerTransport] [prod-resultselasticsearchm-101w] publish_address {10.212.39.91:9200}, bound_addresses {10.212.39.91:9200}
[2022-09-19T13:10:59,212][INFO ][o.e.n.Node               ] [prod-resultselasticsearchm-101w] started
[2022-09-19T13:10:59,993][INFO ][o.e.i.g.DatabaseNodeService] [prod-resultselasticsearchm-101w] retrieve geoip database [GeoLite2-Country.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-4947918882176097480/geoip-databases/ZN56Y0twTkaQOVlD86bPDQ/GeoLite2-Country.mmdb.tmp.gz]
[2022-09-19T13:10:59,994][INFO ][o.e.i.g.DatabaseNodeService] [prod-resultselasticsearchm-101w] retrieve geoip database [GeoLite2-City.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-4947918882176097480/geoip-databases/ZN56Y0twTkaQOVlD86bPDQ/GeoLite2-City.mmdb.tmp.gz]
[2022-09-19T13:10:59,994][INFO ][o.e.i.g.DatabaseNodeService] [prod-resultselasticsearchm-101w] retrieve geoip database [GeoLite2-ASN.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-4947918882176097480/geoip-databases/ZN56Y0twTkaQOVlD86bPDQ/GeoLite2-ASN.mmdb.tmp.gz]
[2022-09-19T13:11:00,244][INFO ][o.e.i.g.DatabaseNodeService] [prod-resultselasticsearchm-101w] successfully reloaded changed geoip database file [/tmp/elasticsearch-4947918882176097480/geoip-databases/ZN56Y0twTkaQOVlD86bPDQ/GeoLite2-Country.mmdb]
[2022-09-19T13:11:00,406][INFO ][o.e.i.g.DatabaseNodeService] [prod-resultselasticsearchm-101w] successfully reloaded changed geoip database file [/tmp/elasticsearch-4947918882176097480/geoip-databases/ZN56Y0twTkaQOVlD86bPDQ/GeoLite2-ASN.mmdb]
[2022-09-19T13:11:01,275][INFO ][o.e.i.g.DatabaseNodeService] [prod-resultselasticsearchm-101w] successfully reloaded changed geoip database file [/tmp/elasticsearch-4947918882176097480/geoip-databases/ZN56Y0twTkaQOVlD86bPDQ/GeoLite2-City.mmdb]
[root@prod-resultselasticsearchm-101w logs]#

I am not familiar with Cerebro, but it could be a generic TLS configuration issue. Without knowing any details, two common TLS client errors are caused by:

  1. Trust verification: The TLS client did not trust the certificate chain of the TLS server.
  2. Hostname verification: The TLS client requested an HTTPS URL address that is missing from the first certificate in the TLS server certificate chain.

For 1, make sure the CA certificate(s) are exported from elastic-certificates.p12, and imported into the Cerebro trust store.

For 2, make sure the TLS server cert contains Subject Alt Name extension with keypairs like dnsName=something or ipAddress=something. Options are FQDN, short hostname, IPv4, or IPv6. dnsName also supports some wildcard use cases. If TLS server is configured with SANs, make sure TLS client is using an address in the cert.

To see an example, click on the lock icon for this web page in your browser. If your browser supports it, you can view the TLS server certificate chain for discuss.elastic.co (or any other public website). This is the server cert exported by my browser for the website.

-----BEGIN CERTIFICATE-----
MIIFKTCCBBGgAwIBAgISBMqKkXELhbtch/OW5QeF0nMiMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMjA5MDcwMDAwMTJaFw0yMjEyMDYwMDAwMTFaMB0xGzAZBgNVBAMT
EmRpc2N1c3MuZWxhc3RpYy5jbzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC
ggEBAKI/MrpFQeuqkMqM8yEjZT1yLGREoVY6MbquetTPfa4sbpqKg8RSbliUzCzb
mRFTZf8djA5H6z+nawygF3sQwtM2Xdhj9ZT27RnD0BwGwK9gPAvFih5c2t5WiIKW
+5VLIneGk2qFxv1tGhuikL+u6AyH0JtFq8I+tzC6klc5HjX5bYlLCjPA7RxXKOJf
DZabiwdnSWPjpChnEGI2Zkne5oVPaHZ8QC7NwU+17IqIcLwegStewkaPESIyNbYp
N3oEx2BR/BTgWCmMPVixGM9pU9C03qiFPimXm+knqgxEKa8C0c5/x3Z6jwj4v2B2
kTxqSId0g4usw7Y0hl7MQLhxHG8CAwEAAaOCAkwwggJIMA4GA1UdDwEB/wQEAwIF
oDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADAd
BgNVHQ4EFgQUwALeAbDmOAXOkpmcCK0+BlFrP74wHwYDVR0jBBgwFoAUFC6zF7dY
VsuuUAlA5h+vnYsUwsYwVQYIKwYBBQUHAQEESTBHMCEGCCsGAQUFBzABhhVodHRw
Oi8vcjMuby5sZW5jci5vcmcwIgYIKwYBBQUHMAKGFmh0dHA6Ly9yMy5pLmxlbmNy
Lm9yZy8wHQYDVR0RBBYwFIISZGlzY3Vzcy5lbGFzdGljLmNvMEwGA1UdIARFMEMw
CAYGZ4EMAQIBMDcGCysGAQQBgt8TAQEBMCgwJgYIKwYBBQUHAgEWGmh0dHA6Ly9j
cHMubGV0c2VuY3J5cHQub3JnMIIBAwYKKwYBBAHWeQIEAgSB9ASB8QDvAHYARqVV
63X6kSAwtaKJafTzfREsQXS+/Um4havy/HD+bUcAAAGDFXY3OgAABAMARzBFAiEA
maNkuGmXedlj48Nv5FoGIlKDkrYFbDGGr5nhEFUdgsACIAQmDPV6CUCHOP+NJHC4
O7FVGT+Mi26UjTa7uTAkZ0DRAHUA36Veq2iCTx9sre64X04+WurNohKkal6OOxLA
IERcKnMAAAGDFXY5AwAABAMARjBEAiBNW7d8udn5oR/pwJ00TpECBMq11qRsvavr
VE0upr1KGAIgKDHZZcSGkGh3RtNPyvBdZUfb/WfR9zyrOSpYv4VL928wDQYJKoZI
hvcNAQELBQADggEBACP9j3svVJpJrQV4lN7bElgB8dKXxDLPBJMh0j612fUoOpgD
NWV2Zdrdbjf4vCdl5GEoE/rsJv2eIg9owrp8wEQjTSQrD8w7eISQjhW/njVJbuJt
ZaYSiNBrBJAQ6GTEy1mDsjHrorI//Yf2qV6DrLsugzdkIVoBug5quHOpL/QPL2kT
5DhjTD4BmI+VZqDG3JlJRR2YL7o8BkSgLPoCvWBu77tTRUBQTHdjudSohcNwZqKm
v5sPV2Z/jkTqOVSJuG0k7traxQ86Z0ZhLkhxL3dZgp6toTKBbp0NtGgLvIhAO64E
1Dyais+jCWPRmznAML0gOkrnKRldrtDWSfxub/I=
-----END CERTIFICATE-----

For this TLS server certificate chain, there are two CAs, and my browser came preinstalled with those CA certs in its truststore because they are showing as trusted. That is sufficient to pass 1.

For the leaf certificate, I exported the cert as PEM and uses an online X.509 cert decoder. I can see it contains a Subject Alt Name extension with dnsName=discuss.elastic.co. That is sufficient to pass 2.

If I were to lookup the IP address of discuss.elastic.co and use that in a URL, the URL https://184.105.176.47/ would pass 1 and fail 2. I tried it in Chrome, and I got an error that the cert is untrusted, because the IP address is not in a SAN ipAddress in the cert.

I am not sure if we are talking about the same thing. I am using the recommended way to create a cert on the stack.

/usr/share/elasticsearch/bin/elasticsearch-certutil cert -out /tmp/elastic-certificates.p12 -pass "" 

then pushing the output cert to all nodes in the stack and restarting the stack itself.

when I do this on other stacks I have implemented in my environment I have had no issues. but for some reason, this one is not working.

I cannot find any documentation on importing the crt to cerebro keystore. we do not use this on other stacks that are working so I am confused.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.