Most simple request:
curl -X GET "http://localhost:9200/_cluster/health?pretty"
The logs:
[2024-04-30T09:08:09,054][INFO ][o.e.n.Node ] [server] stopping ...
[2024-04-30T09:08:09,057][INFO ][o.e.c.f.AbstractFileWatchingService] [server] shutting down watcher thread
[2024-04-30T09:08:09,060][INFO ][o.e.c.f.AbstractFileWatchingService] [server] watcher service stopped
[2024-04-30T09:08:09,061][INFO ][o.e.x.w.WatcherService ] [server] stopping watch service, reason [shutdown initiated]
[2024-04-30T09:08:09,061][INFO ][o.e.x.w.WatcherLifeCycleService] [server] watcher has stopped and shutdown
[2024-04-30T09:08:09,065][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [server] [controller/1843] [Main.cc@176] ML controller exiting
[2024-04-30T09:08:09,066][INFO ][o.e.x.m.p.NativeController] [server] Native controller process has stopped - no new native processes can be started
[2024-04-30T09:08:12,851][INFO ][o.e.n.Node ] [server] stopped
[2024-04-30T09:08:12,851][INFO ][o.e.n.Node ] [server] closing ...
[2024-04-30T09:08:12,889][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [server] evicted [0] entries from cache after reloading database [/tmp/elasticsearch-12221783757989146854/geoip-databases/MOFXJpsyRSi1h-vco7V2Qw/GeoLite2-Country.mmdb]
[2024-04-30T09:08:12,890][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [server] evicted [500] entries from cache after reloading database [/tmp/elasticsearch-12221783757989146854/geoip-databases/MOFXJpsyRSi1h-vco7V2Qw/GeoLite2-ASN.mmdb]
[2024-04-30T09:08:12,890][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [server] evicted [500] entries from cache after reloading database [/tmp/elasticsearch-12221783757989146854/geoip-databases/MOFXJpsyRSi1h-vco7V2Qw/GeoLite2-City.mmdb]
[2024-04-30T09:08:12,929][INFO ][o.e.n.Node ] [server] closed
[2024-04-30T09:08:21,342][INFO ][o.e.n.NativeAccess ] [server] Using [jdk] native provider and native methods for [Linux]
[2024-04-30T09:08:21,805][INFO ][o.a.l.i.v.PanamaVectorizationProvider] [server] Java vector incubator API enabled; uses preferredBitSize=256; FMA enabled
[2024-04-30T09:08:22,655][INFO ][o.e.n.Node ] [server] version[8.13.2], pid[2997142], build[rpm/16cc90cd2d08a3147ce02b07e50894bc060a4cbf/2024-04-05T14:45:26.420424304Z], OS[Linux/5.14.0-437.el9.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/21.0.2/21.0.2+13-58]
[2024-04-30T09:08:22,656][INFO ][o.e.n.Node ] [server] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2024-04-30T09:08:22,656][INFO ][o.e.n.Node ] [server] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=org.elasticsearch.preallocate, --enable-native-access=org.elasticsearch.nativeaccess, -XX:ReplayDataFile=/var/log/elasticsearch/replay_pid%p.log, -Des.distribution.type=rpm, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-11523029729499202865, --add-modules=jdk.incubator.vector, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,level,pid,tags:filecount=32,filesize=64m, -Xms15920m, -Xmx15920m, -XX:MaxDirectMemorySize=8346664960, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=25, --module-path=/usr/share/elasticsearch/lib, --add-modules=jdk.net, --add-modules=ALL-MODULE-PATH, -Djdk.module.main=org.elasticsearch.server]
[2024-04-30T09:08:22,657][INFO ][o.e.n.Node ] [server] Default Locale [en_US]
[2024-04-30T09:08:26,890][INFO ][o.e.p.PluginsService ] [server] loaded module [repository-url]
[2024-04-30T09:08:26,891][INFO ][o.e.p.PluginsService ] [server] loaded module [rest-root]
[2024-04-30T09:08:26,891][INFO ][o.e.p.PluginsService ] [server] loaded module [x-pack-core]
[2024-04-30T09:08:26,891][INFO ][o.e.p.PluginsService ] [server] loaded module [x-pack-redact]
[2024-04-30T09:08:26,891][INFO ][o.e.p.PluginsService ] [server] loaded module [ingest-user-agent]
[2024-04-30T09:08:26,891][INFO ][o.e.p.PluginsService ] [server] loaded module [x-pack-async-search]
[2024-04-30T09:08:26,891][INFO ][o.e.p.PluginsService ] [server] loaded module [x-pack-monitoring]
[2024-04-30T09:08:26,891][INFO ][o.e.p.PluginsService ] [server] loaded module [repository-s3]
[2024-04-30T09:08:26,891][INFO ][o.e.p.PluginsService ] [server] loaded module [x-pack-analytics]
[2024-04-30T09:08:26,891][INFO ][o.e.p.PluginsService ] [server] loaded module [x-pack-ent-search]
[2024-04-30T09:08:26,892][INFO ][o.e.p.PluginsService ] [server] loaded module [x-pack-autoscaling]
[2024-04-30T09:08:26,892][INFO ][o.e.p.PluginsService ] [server] loaded module [lang-painless]
[2024-04-30T09:08:26,892][INFO ][o.e.p.PluginsService ] [server] loaded module [x-pack-ml]
[2024-04-30T09:08:26,892][INFO ][o.e.p.PluginsService ] [server] loaded module [lang-mustache]
[2024-04-30T09:08:26,892][INFO ][o.e.p.PluginsService ] [server] loaded module [legacy-geo]
[2024-04-30T09:08:26,892][INFO ][o.e.p.PluginsService ] [server] loaded module [x-pack-ql]
[2024-04-30T09:08:26,892][INFO ][o.e.p.PluginsService ] [server] loaded module [rank-rrf]
[2024-04-30T09:08:26,892][INFO ][o.e.p.PluginsService ] [server] loaded module [analysis-common]
[2024-04-30T09:08:26,892][INFO ][o.e.p.PluginsService ] [server] loaded module [health-shards-availability]
[2024-04-30T09:08:26,893][INFO ][o.e.p.PluginsService ] [server] loaded module [transport-netty4]
[2024-04-30T09:08:26,893][INFO ][o.e.p.PluginsService ] [server] loaded module [aggregations]
[2024-04-30T09:08:26,900][INFO ][o.e.p.PluginsService ] [server] loaded module [ingest-common]
[2024-04-30T09:08:26,901][INFO ][o.e.p.PluginsService ] [server] loaded module [frozen-indices]
[2024-04-30T09:08:26,901][INFO ][o.e.p.PluginsService ] [server] loaded module [x-pack-identity-provider]
[2024-04-30T09:08:26,901][INFO ][o.e.p.PluginsService ] [server] loaded module [x-pack-text-structure]
[2024-04-30T09:08:26,901][INFO ][o.e.p.PluginsService ] [server] loaded module [x-pack-shutdown]
[2024-04-30T09:08:26,901][INFO ][o.e.p.PluginsService ] [server] loaded module [snapshot-repo-test-kit]
[2024-04-30T09:08:26,901][INFO ][o.e.p.PluginsService ] [server] loaded module [ml-package-loader]
[2024-04-30T09:08:26,901][INFO ][o.e.p.PluginsService ] [server] loaded module [kibana]
[2024-04-30T09:08:26,902][INFO ][o.e.p.PluginsService ] [server] loaded module [constant-keyword]
[2024-04-30T09:08:26,902][INFO ][o.e.p.PluginsService ] [server] loaded module [x-pack-logstash]
[2024-04-30T09:08:26,902][INFO ][o.e.p.PluginsService ] [server] loaded module [x-pack-graph]
[2024-04-30T09:08:26,902][INFO ][o.e.p.PluginsService ] [server] loaded module [x-pack-ccr]
[2024-04-30T09:08:26,902][INFO ][o.e.p.PluginsService ] [server] loaded module [x-pack-esql]
[2024-04-30T09:08:26,902][INFO ][o.e.p.PluginsService ] [server] loaded module [parent-join]
[2024-04-30T09:08:26,902][INFO ][o.e.p.PluginsService ] [server] loaded module [counted-keyword]
[2024-04-30T09:08:26,902][INFO ][o.e.p.PluginsService ] [server] loaded module [x-pack-enrich]
[2024-04-30T09:08:26,902][INFO ][o.e.p.PluginsService ] [server] loaded module [repositories-metering-api]
[2024-04-30T09:08:26,903][INFO ][o.e.p.PluginsService ] [server] loaded module [transform]
[2024-04-30T09:08:26,903][INFO ][o.e.p.PluginsService ] [server] loaded module [repository-azure]
[2024-04-30T09:08:26,903][INFO ][o.e.p.PluginsService ] [server] loaded module [repository-gcs]
[2024-04-30T09:08:26,903][INFO ][o.e.p.PluginsService ] [server] loaded module [spatial]
[2024-04-30T09:08:26,903][INFO ][o.e.p.PluginsService ] [server] loaded module [mapper-version]
[2024-04-30T09:08:26,906][INFO ][o.e.p.PluginsService ] [server] loaded module [apm]
[2024-04-30T09:08:26,906][INFO ][o.e.p.PluginsService ] [server] loaded module [mapper-extras]
[2024-04-30T09:08:26,906][INFO ][o.e.p.PluginsService ] [server] loaded module [x-pack-rollup]
[2024-04-30T09:08:26,906][INFO ][o.e.p.PluginsService ] [server] loaded module [percolator]
[2024-04-30T09:08:26,906][INFO ][o.e.p.PluginsService ] [server] loaded module [data-streams]
[2024-04-30T09:08:26,906][INFO ][o.e.p.PluginsService ] [server] loaded module [x-pack-stack]
[2024-04-30T09:08:26,907][INFO ][o.e.p.PluginsService ] [server] loaded module [reindex]
[2024-04-30T09:08:26,909][INFO ][o.e.p.PluginsService ] [server] loaded module [rank-eval]
[2024-04-30T09:08:26,909][INFO ][o.e.p.PluginsService ] [server] loaded module [systemd]
[2024-04-30T09:08:26,909][INFO ][o.e.p.PluginsService ] [server] loaded module [x-pack-security]
[2024-04-30T09:08:26,909][INFO ][o.e.p.PluginsService ] [server] loaded module [blob-cache]
[2024-04-30T09:08:26,910][INFO ][o.e.p.PluginsService ] [server] loaded module [searchable-snapshots]
[2024-04-30T09:08:26,910][INFO ][o.e.p.PluginsService ] [server] loaded module [x-pack-slm]
[2024-04-30T09:08:26,910][INFO ][o.e.p.PluginsService ] [server] loaded module [snapshot-based-recoveries]
[2024-04-30T09:08:26,910][INFO ][o.e.p.PluginsService ] [server] loaded module [x-pack-watcher]
[2024-04-30T09:08:26,910][INFO ][o.e.p.PluginsService ] [server] loaded module [old-lucene-versions]
[2024-04-30T09:08:26,910][INFO ][o.e.p.PluginsService ] [server] loaded module [x-pack-ilm]
[2024-04-30T09:08:26,910][INFO ][o.e.p.PluginsService ] [server] loaded module [x-pack-voting-only-node]
[2024-04-30T09:08:26,910][INFO ][o.e.p.PluginsService ] [server] loaded module [x-pack-inference]
[2024-04-30T09:08:26,910][INFO ][o.e.p.PluginsService ] [server] loaded module [x-pack-deprecation]
[2024-04-30T09:08:26,910][INFO ][o.e.p.PluginsService ] [server] loaded module [x-pack-fleet]
[2024-04-30T09:08:26,910][INFO ][o.e.p.PluginsService ] [server] loaded module [x-pack-aggregate-metric]
[2024-04-30T09:08:26,910][INFO ][o.e.p.PluginsService ] [server] loaded module [x-pack-downsample]
[2024-04-30T09:08:26,911][INFO ][o.e.p.PluginsService ] [server] loaded module [x-pack-profiling]
[2024-04-30T09:08:26,911][INFO ][o.e.p.PluginsService ] [server] loaded module [ingest-geoip]
[2024-04-30T09:08:26,911][INFO ][o.e.p.PluginsService ] [server] loaded module [x-pack-write-load-forecaster]
[2024-04-30T09:08:26,911][INFO ][o.e.p.PluginsService ] [server] loaded module [search-business-rules]
[2024-04-30T09:08:26,911][INFO ][o.e.p.PluginsService ] [server] loaded module [ingest-attachment]
[2024-04-30T09:08:26,911][INFO ][o.e.p.PluginsService ] [server] loaded module [wildcard]
[2024-04-30T09:08:26,911][INFO ][o.e.p.PluginsService ] [server] loaded module [x-pack-apm-data]
[2024-04-30T09:08:26,911][INFO ][o.e.p.PluginsService ] [server] loaded module [x-pack-sql]
[2024-04-30T09:08:26,911][INFO ][o.e.p.PluginsService ] [server] loaded module [unsigned-long]
[2024-04-30T09:08:26,911][INFO ][o.e.p.PluginsService ] [server] loaded module [runtime-fields-common]
[2024-04-30T09:08:26,911][INFO ][o.e.p.PluginsService ] [server] loaded module [x-pack-async]
[2024-04-30T09:08:26,912][INFO ][o.e.p.PluginsService ] [server] loaded module [vector-tile]
[2024-04-30T09:08:26,912][INFO ][o.e.p.PluginsService ] [server] loaded module [lang-expression]
[2024-04-30T09:08:26,912][INFO ][o.e.p.PluginsService ] [server] loaded module [x-pack-eql]
[2024-04-30T09:08:28,140][INFO ][o.e.e.NodeEnvironment ] [server] using [1] data paths, mounts [[/data (172.20.5.101:/data)]], net usable_space [554gb], net total_space [589.5gb], types [nfs4]
[2024-04-30T09:08:28,141][INFO ][o.e.e.NodeEnvironment ] [server] heap size [15.5gb], compressed ordinary object pointers [true]
[2024-04-30T09:08:30,509][INFO ][o.e.n.Node ] [server] node name [server], node ID [MOFXJpsyRSi1h-vco7V2Qw], cluster name [companycluster], roles [data_frozen, ingest, data_cold, data, remote_cluster_client, master, data_warm, data_content, transform, data_hot, ml]
[2024-04-30T09:08:36,237][INFO ][o.e.f.FeatureService ] [server] Registered local node features [data_stream.rollover.lazy, desired_node.version_deprecated, features_supported, health.dsl.info, health.extended_repository_indicator, usage.data_tiers.precalculate_stats]
[2024-04-30T09:08:36,746][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [server] [controller/2997220] [Main.cc@123] controller (64 bit): Version 8.13.2 (Build fdd7177d8c1325) Copyright (c) 2024 Elasticsearch BV
[2024-04-30T09:08:37,022][INFO ][o.e.t.a.APM ] [server] Sending apm metrics is disabled
[2024-04-30T09:08:37,022][INFO ][o.e.t.a.APM ] [server] Sending apm tracing is disabled
[2024-04-30T09:08:37,047][INFO ][o.e.x.s.Security ] [server] Security is enabled
[2024-04-30T09:08:37,339][INFO ][o.e.x.s.a.s.FileRolesStore] [server] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2024-04-30T09:08:37,775][INFO ][o.e.x.s.InitialNodeSecurityAutoConfiguration] [server] Auto-configuration will not generate a password for the elastic built-in superuser, as we cannot determine if there is a terminal attached to the elasticsearch process. You can use the `bin/elasticsearch-reset-password` tool to set the password for the elastic user.
[2024-04-30T09:08:38,078][INFO ][o.e.x.w.Watcher ] [server] Watcher initialized components at 2024-04-30T07:08:38.078Z
[2024-04-30T09:08:38,137][INFO ][o.e.x.p.ProfilingPlugin ] [server] Profiling is enabled
[2024-04-30T09:08:38,171][INFO ][o.e.x.p.ProfilingPlugin ] [server] profiling index templates will not be installed or reinstalled
[2024-04-30T09:08:38,175][INFO ][o.e.x.a.APMPlugin ] [server] APM ingest plugin is disabled
[2024-04-30T09:08:38,623][INFO ][o.e.t.n.NettyAllocator ] [server] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=8mb}]
[2024-04-30T09:08:38,648][INFO ][o.e.i.r.RecoverySettings ] [server] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]
[2024-04-30T09:08:38,689][INFO ][o.e.d.DiscoveryModule ] [server] using discovery type [multi-node] and seed hosts providers [settings]
[2024-04-30T09:08:40,686][INFO ][o.e.n.Node ] [server] initialized
[2024-04-30T09:08:40,687][INFO ][o.e.n.Node ] [server] starting ...
[2024-04-30T09:08:40,793][INFO ][o.e.x.s.c.f.PersistentCache] [server] persistent cache index loaded
[2024-04-30T09:08:40,803][INFO ][o.e.x.d.l.DeprecationIndexingComponent] [server] deprecation component started
[2024-04-30T09:08:40,911][INFO ][o.e.t.TransportService ] [server] publish_address {172.20.5.100:9300}, bound_addresses {0.0.0.0:9300}
[2024-04-30T09:08:43,256][INFO ][o.e.b.BootstrapChecks ] [server] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2024-04-30T09:08:43,275][WARN ][o.e.c.c.ClusterBootstrapService] [server] this node is locked into cluster UUID [H2A9Q-BGTmuHlZBMYT9VFQ] but [cluster.initial_master_nodes] is set to [server]; remove this setting to avoid possible data loss caused by subsequent cluster bootstrap attempts; for further information see https://www.elastic.co/guide/en/elasticsearch/reference/8.13/important-settings.html#initial_master_nodes
[2024-04-30T09:08:43,473][INFO ][o.e.c.s.MasterService ] [server] elected-as-master ([1] nodes joined in term 7)[_FINISH_ELECTION_, {server}{MOFXJpsyRSi1h-vco7V2Qw}{KBDzbiypQriVhlgKwWJwGw}{server}{172.20.5.100}{172.20.5.100:9300}{cdfhilmrstw}{8.13.2}{7000099-8503000} completing election], term: 7, version: 2195, delta: master node changed {previous [], current [{server}{MOFXJpsyRSi1h-vco7V2Qw}{KBDzbiypQriVhlgKwWJwGw}{server}{172.20.5.100}{172.20.5.100:9300}{cdfhilmrstw}{8.13.2}{7000099-8503000}]}
[2024-04-30T09:08:43,837][INFO ][o.e.c.s.ClusterApplierService] [server] master node changed {previous [], current [{server}{MOFXJpsyRSi1h-vco7V2Qw}{KBDzbiypQriVhlgKwWJwGw}{server}{172.20.5.100}{172.20.5.100:9300}{cdfhilmrstw}{8.13.2}{7000099-8503000}]}, term: 7, version: 2195, reason: Publication{term=7, version=2195}
[2024-04-30T09:08:43,914][INFO ][o.e.c.f.AbstractFileWatchingService] [server] starting file watcher ...
[2024-04-30T09:08:43,928][INFO ][o.e.c.f.AbstractFileWatchingService] [server] file settings service up and running [tid=57]
[2024-04-30T09:08:43,928][INFO ][o.e.c.c.NodeJoinExecutor ] [server] node-join: [{server}{MOFXJpsyRSi1h-vco7V2Qw}{KBDzbiypQriVhlgKwWJwGw}{server}{172.20.5.100}{172.20.5.100:9300}{cdfhilmrstw}{8.13.2}{7000099-8503000}] with reason [completing election]
[2024-04-30T09:08:43,932][INFO ][o.e.h.AbstractHttpServerTransport] [server] publish_address {172.20.5.100:9200}, bound_addresses {0.0.0.0:9200}
[2024-04-30T09:08:43,962][INFO ][o.e.n.Node ] [server] started {server}{MOFXJpsyRSi1h-vco7V2Qw}{KBDzbiypQriVhlgKwWJwGw}{server}{172.20.5.100}{172.20.5.100:9300}{cdfhilmrstw}{8.13.2}{7000099-8503000}{ml.allocated_processors_double=4.0, ml.allocated_processors=4, ml.machine_memory=33387298816, transform.config_version=10.0.0, xpack.installed=true, ml.config_version=12.0.0, ml.max_jvm_size=16693329920}
[2024-04-30T09:08:43,989][INFO ][o.e.c.s.ClusterSettings ] [server] updating [xpack.monitoring.collection.enabled] from [false] to [true]
[2024-04-30T09:08:45,378][INFO ][o.e.x.s.a.RealmsAuthenticator] [server] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2024-04-30T09:08:45,374][INFO ][o.e.x.s.a.RealmsAuthenticator] [server] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2024-04-30T09:08:46,596][INFO ][o.e.x.s.a.RealmsAuthenticator] [server] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2024-04-30T09:08:46,596][INFO ][o.e.x.s.a.RealmsAuthenticator] [server] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2024-04-30T09:08:47,055][ERROR][o.e.x.s.a.e.ReservedRealm] [server] failed to retrieve password hash for reserved user [elastic]
org.elasticsearch.action.UnavailableShardsException: at least one primary shard for the index [.security-7] is unavailable
at org.elasticsearch.xpack.security.support.SecurityIndexManager.getUnavailableReason(SecurityIndexManager.java:178) ~[?:?]
at org.elasticsearch.xpack.security.authc.esnative.NativeUsersStore.getReservedUserInfo(NativeUsersStore.java:640) ~[?:?]
at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.getUserInfo(ReservedRealm.java:272) ~[?:?]
at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.doAuthenticate(ReservedRealm.java:136) ~[?:?]
at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:200) ~[?:?]
at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:105) ~[?:?]
at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$4(RealmsAuthenticator.java:170) ~[?:?]
at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117) ~[?:?]
at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.consumeToken(RealmsAuthenticator.java:263) ~[?:?]
at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.authenticate(RealmsAuthenticator.java:106) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:167) ~[?:?]
at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:146) ~[?:?]
at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:146) ~[?:?]
at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:146) ~[?:?]
at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.doAuthenticate(AuthenticatorChain.java:125) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticate(AuthenticatorChain.java:95) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:264) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:152) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:130) ~[?:?]
at org.elasticsearch.xpack.security.Security.lambda$getHttpTransports$35(Security.java:1758) ~[?:?]
at org.elasticsearch.xpack.security.Security.lambda$getHttpServerTransportWithHeadersValidator$38(Security.java:1815) ~[?:?]
at org.elasticsearch.http.netty4.internal.HttpHeadersAuthenticatorUtils.lambda$getValidatorInboundHandler$2(HttpHeadersAuthenticatorUtils.java:47) ~[?:?]
at org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.requestStart(Netty4HttpHeaderValidator.java:126) ~[?:?]
at org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.channelRead(Netty4HttpHeaderValidator.java:61) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1383) ~[?:?]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246) ~[?:?]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:1583) ~[?:?]
[2024-04-30T09:08:47,062][INFO ][o.e.x.s.a.RealmsAuthenticator] [server] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2024-04-30T09:08:47,098][INFO ][o.e.x.s.a.Realms ] [server] license mode is [basic], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]
[2024-04-30T09:08:47,110][INFO ][o.e.l.ClusterStateLicenseService] [server] license [3575729b-d36a-431e-bb91-0937d6539cf3] mode [basic] - valid
[2024-04-30T09:08:47,118][INFO ][o.e.g.GatewayService ] [server] recovered [79] indices into cluster_state
[2024-04-30T09:08:47,769][INFO ][o.e.h.n.s.HealthNodeTaskExecutor] [server] Node [{server}{MOFXJpsyRSi1h-vco7V2Qw}] is selected as the current health node.
[2024-04-30T09:08:47,770][ERROR][o.e.i.g.GeoIpDownloader ] [server] exception during geoip databases update
org.elasticsearch.ElasticsearchException: not all primary shards of [.geoip_databases] index are active
at org.elasticsearch.ingest.geoip.GeoIpDownloader.updateDatabases(GeoIpDownloader.java:131) ~[?:?]
at org.elasticsearch.ingest.geoip.GeoIpDownloader.runDownloader(GeoIpDownloader.java:279) ~[?:?]
at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:160) ~[?:?]
at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:59) ~[?:?]
at org.elasticsearch.persistent.NodePersistentTasksExecutor$1.doRun(NodePersistentTasksExecutor.java:34) ~[elasticsearch-8.13.2.jar:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984) ~[elasticsearch-8.13.2.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.13.2.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1583) ~[?:?]
[2024-04-30T09:08:47,894][ERROR][o.e.x.s.a.e.ReservedRealm] [server] failed to retrieve password hash for reserved user [elastic]
org.elasticsearch.action.UnavailableShardsException: at least one primary shard for the index [.security-7] is unavailable
at org.elasticsearch.xpack.security.support.SecurityIndexManager.getUnavailableReason(SecurityIndexManager.java:178) ~[?:?]
at org.elasticsearch.xpack.security.authc.esnative.NativeUsersStore.getReservedUserInfo(NativeUsersStore.java:640) ~[?:?]
at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.getUserInfo(ReservedRealm.java:272) ~[?:?]
at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.doAuthenticate(ReservedRealm.java:136) ~[?:?]
at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:200) ~[?:?]
at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:105) ~[?:?]
at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$4(RealmsAuthenticator.java:170) ~[?:?]
at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117) ~[?:?]
at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.consumeToken(RealmsAuthenticator.java:263) ~[?:?]
at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.authenticate(RealmsAuthenticator.java:106) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:167) ~[?:?]
at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:146) ~[?:?]
at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:146) ~[?:?]
at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:146) ~[?:?]
at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.doAuthenticate(AuthenticatorChain.java:125) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticate(AuthenticatorChain.java:95) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:264) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:152) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:130) ~[?:?]
at org.elasticsearch.xpack.security.Security.lambda$getHttpTransports$35(Security.java:1758) ~[?:?]
at org.elasticsearch.xpack.security.Security.lambda$getHttpServerTransportWithHeadersValidator$38(Security.java:1815) ~[?:?]
at org.elasticsearch.http.netty4.internal.HttpHeadersAuthenticatorUtils.lambda$getValidatorInboundHandler$2(HttpHeadersAuthenticatorUtils.java:47) ~[?:?]
at org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.requestStart(Netty4HttpHeaderValidator.java:126) ~[?:?]
at org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.channelRead(Netty4HttpHeaderValidator.java:61) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1383) ~[?:?]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246) ~[?:?]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:1583) ~[?:?]
[2024-04-30T09:08:47,897][INFO ][o.e.x.s.a.RealmsAuthenticator] [server] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2024-04-30T09:08:48,538][ERROR][o.e.x.s.a.e.ReservedRealm] [server] failed to retrieve password hash for reserved user [elastic]
org.elasticsearch.action.UnavailableShardsException: at least one primary shard for the index [.security-7] is unavailable
at org.elasticsearch.xpack.security.support.SecurityIndexManager.getUnavailableReason(SecurityIndexManager.java:178) ~[?:?]
at org.elasticsearch.xpack.security.authc.esnative.NativeUsersStore.getReservedUserInfo(NativeUsersStore.java:640) ~[?:?]
at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.getUserInfo(ReservedRealm.java:272) ~[?:?]
at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.doAuthenticate(ReservedRealm.java:136) ~[?:?]
at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:200) ~[?:?]
at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:105) ~[?:?]
at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$4(RealmsAuthenticator.java:170) ~[?:?]
at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117) ~[?:?]
at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.consumeToken(RealmsAuthenticator.java:263) ~[?:?]
at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.authenticate(RealmsAuthenticator.java:106) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:167) ~[?:?]
at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:146) ~[?:?]
at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:146) ~[?:?]
at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:146) ~[?:?]
at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.doAuthenticate(AuthenticatorChain.java:125) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticate(AuthenticatorChain.java:95) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:264) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:152) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:130) ~[?:?]
at org.elasticsearch.xpack.security.Security.lambda$getHttpTransports$35(Security.java:1758) ~[?:?]
at org.elasticsearch.xpack.security.Security.lambda$getHttpServerTransportWithHeadersValidator$38(Security.java:1815) ~[?:?]
at org.elasticsearch.http.netty4.internal.HttpHeadersAuthenticatorUtils.lambda$getValidatorInboundHandler$2(HttpHeadersAuthenticatorUtils.java:47) ~[?:?]
at org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.requestStart(Netty4HttpHeaderValidator.java:126) ~[?:?]
at org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.channelRead(Netty4HttpHeaderValidator.java:61) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1383) ~[?:?]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246) ~[?:?]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:1583) ~[?:?]
[2024-04-30T09:08:48,542][INFO ][o.e.x.s.a.RealmsAuthenticator] [server] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2024-04-30T09:08:50,527][INFO ][o.e.i.g.DatabaseNodeService] [server] successfully loaded geoip database file [GeoLite2-Country.mmdb]
[2024-04-30T09:08:50,884][INFO ][o.e.i.g.DatabaseNodeService] [server] successfully loaded geoip database file [GeoLite2-ASN.mmdb]
[2024-04-30T09:08:52,543][INFO ][o.e.i.g.DatabaseNodeService] [server] successfully loaded geoip database file [GeoLite2-City.mmdb]
[2024-04-30T09:08:54,505][WARN ][o.e.h.n.Netty4HttpServerTransport] [server] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:59058}
[2024-04-30T09:09:02,612][INFO ][o.e.c.r.a.AllocationService] [server] current.health="YELLOW" message="Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.kibana-observability-ai-assistant-conversations-000001][0], [.apm-source-map][0]]])." previous.health="RED" reason="shards started [[.kibana-observability-ai-assistant-conversations-000001][0], [.apm-source-map][0]]"
[2024-04-30T09:09:03,495][WARN ][o.e.h.n.Netty4HttpServerTransport] [server] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:36884}
[2024-04-30T09:09:33,698][WARN ][o.e.h.n.Netty4HttpServerTransport] [server] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:40592}
[2024-04-30T09:09:45,251][WARN ][o.e.h.n.Netty4HttpServerTransport] [server] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:52254}
[2024-04-30T09:09:50,613][WARN ][o.e.h.n.Netty4HttpServerTransport] [server] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:52260}
[2024-04-30T09:10:28,822][WARN ][o.e.h.n.Netty4HttpServerTransport] [server] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:59108}
[2024-04-30T09:10:42,834][WARN ][o.e.h.n.Netty4HttpServerTransport] [server] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:42186}
[2024-04-30T09:10:49,729][WARN ][o.e.h.n.Netty4HttpServerTransport] [server] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:59716}
[2024-04-30T09:11:18,576][WARN ][o.e.h.n.Netty4HttpServerTransport] [server] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:55112}
[2024-04-30T09:11:38,137][WARN ][o.e.h.n.Netty4HttpServerTransport] [server] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:58982}
[2024-04-30T09:11:46,895][WARN ][o.e.h.n.Netty4HttpServerTransport] [server] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:56032}
[2024-04-30T09:12:17,274][WARN ][o.e.h.n.Netty4HttpServerTransport] [server] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:48480}
[2024-04-30T09:12:18,501][WARN ][o.e.h.n.Netty4HttpServerTransport] [server] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:48492}
[2024-04-30T09:12:19,885][WARN ][o.e.h.n.Netty4HttpServerTransport] [server] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:48508}
[2024-04-30T09:12:52,404][WARN ][o.e.h.n.Netty4HttpServerTransport] [server] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:58844}
[2024-04-30T09:13:12,560][WARN ][o.e.h.n.Netty4HttpServerTransport] [server] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:57022}
[2024-04-30T09:13:17,952][WARN ][o.e.h.n.Netty4HttpServerTransport] [server] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:38424}
[2024-04-30T09:13:43,949][WARN ][o.e.h.n.Netty4HttpServerTransport] [server] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:40972}
[2024-04-30T09:13:44,100][INFO ][o.e.c.s.IndexScopedSettings] [server] [.ds-metrics-elastic_agent.metricbeat-default-2024.04.24-000001] updating [index.time_series.end_time] from [2024-04-30T07:39:13.000Z] to [2024-04-30T07:48:43.000Z]
[2024-04-30T09:13:44,101][INFO ][o.e.c.s.IndexScopedSettings] [server] [.ds-metrics-system.filesystem-default-2024.04.24-000001] updating [index.time_series.end_time] from [2024-04-30T07:39:13.000Z] to [2024-04-30T07:48:43.000Z]
[2024-04-30T09:13:44,101][INFO ][o.e.c.s.IndexScopedSettings] [server] [.ds-metrics-system.cpu-default-2024.04.24-000001] updating [index.time_series.end_time] from [2024-04-30T07:39:13.000Z] to [2024-04-30T07:48:43.000Z]
[2024-04-30T09:13:44,102][INFO ][o.e.c.s.IndexScopedSettings] [server] [.ds-metrics-elastic_agent.elastic_agent-default-2024.04.24-000001] updating [index.time_series.end_time] from [2024-04-30T07:39:13.000Z] to [2024-04-30T07:48:43.000Z]
[2024-04-30T09:13:44,103][INFO ][o.e.c.s.IndexScopedSettings] [server] [.ds-metrics-elastic_agent.filebeat-default-2024.04.24-000001] updating [index.time_series.end_time] from [2024-04-30T07:39:13.000Z] to [2024-04-30T07:48:43.000Z]
[2024-04-30T09:13:44,103][INFO ][o.e.c.s.IndexScopedSettings] [server] [.ds-metrics-system.process.summary-default-2024.04.24-000001] updating [index.time_series.end_time] from [2024-04-30T07:39:13.000Z] to [2024-04-30T07:48:43.000Z]
[2024-04-30T09:13:44,104][INFO ][o.e.c.s.IndexScopedSettings] [server] [.ds-metrics-system.fsstat-default-2024.04.24-000001] updating [index.time_series.end_time] from [2024-04-30T07:39:13.000Z] to [2024-04-30T07:48:43.000Z]
[2024-04-30T09:13:44,104][INFO ][o.e.c.s.IndexScopedSettings] [server] [.ds-metrics-system.uptime-default-2024.04.24-000001] updating [index.time_series.end_time] from [2024-04-30T07:39:13.000Z] to [2024-04-30T07:48:43.000Z]
[2024-04-30T09:13:44,105][INFO ][o.e.c.s.IndexScopedSettings] [server] [.ds-metrics-system.socket_summary-default-2024.04.24-000001] updating [index.time_series.end_time] from [2024-04-30T07:39:13.000Z] to [2024-04-30T07:48:43.000Z]
[2024-04-30T09:13:44,105][INFO ][o.e.c.s.IndexScopedSettings] [server] [.ds-metrics-system.diskio-default-2024.04.24-000001] updating [index.time_series.end_time] from [2024-04-30T07:39:13.000Z] to [2024-04-30T07:48:43.000Z]
[2024-04-30T09:13:44,106][INFO ][o.e.c.s.IndexScopedSettings] [server] [.ds-metrics-system.memory-default-2024.04.24-000001] updating [index.time_series.end_time] from [2024-04-30T07:39:13.000Z] to [2024-04-30T07:48:43.000Z]
[2024-04-30T09:13:44,106][INFO ][o.e.c.s.IndexScopedSettings] [server] [.ds-metrics-windows.service-default-2024.04.25-000001] updating [index.time_series.end_time] from [2024-04-30T07:39:13.000Z] to [2024-04-30T07:48:43.000Z]
[2024-04-30T09:13:44,106][INFO ][o.e.c.s.IndexScopedSettings] [server] [.ds-metrics-system.load-default-2024.04.24-000001] updating [index.time_series.end_time] from [2024-04-30T07:39:13.000Z] to [2024-04-30T07:48:43.000Z]
[2024-04-30T09:13:44,107][INFO ][o.e.c.s.IndexScopedSettings] [server] [.ds-metrics-system.process-default-2024.04.24-000001] updating [index.time_series.end_time] from [2024-04-30T07:39:13.000Z] to [2024-04-30T07:48:43.000Z]
[2024-04-30T09:13:44,107][INFO ][o.e.c.s.IndexScopedSettings] [server] [.ds-metrics-system.network-default-2024.04.24-000001] updating [index.time_series.end_time] from [2024-04-30T07:39:13.000Z] to [2024-04-30T07:48:43.000Z]
[2024-04-30T09:13:48,729][WARN ][o.e.h.n.Netty4HttpServerTransport] [server] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:40976}
[2024-04-30T09:14:06,050][WARN ][o.e.h.n.Netty4HttpServerTransport] [server] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:48668}
I enter Kibana and logs SEEM to be getting collected but that health check should be working