Kibana server is not ready yet and curl: (52) Empty reply from server

Elastic Stack Elasticsearch
1

Hello

Im getting these two error messages

Kibana server is not ready yet

This one appears when I am going to the Kibana page. I checked that all services are up and running so that is OK.

I ran a

curl -XGET 'localhost:9200/_cluster/health?pretty'

and I get

curl: (52) Empty reply from server

So I went into /elasticsearch.yml and set xpack.security.enabled: false

I do now get a proper reply from curl BUT I still get Kibana server is not ready yet

How can I fix it?

Looking at elasticsearch logs, no errors or anything that jumps at me.

Kibana shows this:

"ecs": {
"version": "8.11.0"
},
"@timestamp": "2024-04-23T18:11:27.700+02:00",
"message": "Unable to retrieve version information from Elasticsearch nodes. connect ECONNREFUSED 172.20.5.100:9200",
"log": {
"level": "ERROR",
"logger": "elasticsearch-service"
},
"process": {
"pid": 797,
"uptime": 260.436430754
},
"trace": {
"id": "3cd91ae6e4bb9f10e8e3111590db2e19"
},
"transaction": {
"id": "ff91294db4853f27"
}
}
{
"service": {
"node": {
"roles": [
"background_tasks",
"ui"
]
}
},
"ecs": {
"version": "8.11.0"
},
"@timestamp": "2024-04-23T18:12:05.263+02:00",
"message": "Unable to retrieve version information from Elasticsearch nodes. write EPROTO 40A8C62D737F0000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:354:\n",
"log": {
"level": "ERROR",
"logger": "elasticsearch-service"
},
"process": {
"pid": 797,
"uptime": 297.998377823
},
"trace": {
"id": "3cd91ae6e4bb9f10e8e3111590db2e19"
},
"transaction": {
"id": "ff91294db4853f27"
}
}

However:

lsof -i -P -n | grep LISTEN
systemd 1 root 34u IPv4 2491 0t0 TCP *:111 (LISTEN)
systemd 1 root 36u IPv6 2497 0t0 TCP *:111 (LISTEN)
rpcbind 703 rpc 4u IPv4 2491 0t0 TCP *:111 (LISTEN)
rpcbind 703 rpc 6u IPv6 2497 0t0 TCP *:111 (LISTEN)
sshd 772 root 3u IPv4 19479 0t0 TCP *:22 (LISTEN)
sshd 772 root 4u IPv6 19481 0t0 TCP *:22 (LISTEN)
node 797 kibana 18u IPv4 23300 0t0 TCP *:5601 (LISTEN)
java 3533 elasticsearch 446u IPv6 34821 0t0 TCP *:9300 (LISTEN)
java 3533 elasticsearch 451u IPv6 30646 0t0 TCP *:9200 (LISTEN)

I dont think Logstash logs are needed; If so ask.

This is all installed from repositories.

2

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

This is the icon to use if you are not using markdown format:

image
image764×92 16.4 KB

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.
Please update your post.

Also could you provide the full Elasticsearch logs?

3

This would be a great use for AI; It automatically formats the code....

Anyways, the Elasticsearch logs? Sure.

{"@timestamp":"2024-04-23T22:00:10.297Z", "log.level": "INFO", "message":"[.ds-packetbeat-8.13.2-2024.04.23-000001/dEK4VvXGRR29KRwh3zKYIg] update_mapping [_doc]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[Server][masterService#updateTask][T#71]","log.logger":"org.elasticsearch.cluster.metadata.MetadataMappingService","elasticsearch.cluster.uuid":"TKJOT3aMSbmcJnMlEu-_Vg","elasticsearch.node.id":"JNoURkKaRC-n-WbKLRRb1A","elasticsearch.node.name":"Server","elasticsearch.cluster.name":"companyElasticCluster"}
{"@timestamp":"2024-04-23T22:58:00.001Z", "log.level": "INFO", "message":"triggering scheduled [ML] maintenance tasks", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[Server][generic][T#3]","log.logger":"org.elasticsearch.xpack.ml.MlDailyMaintenanceService","elasticsearch.cluster.uuid":"TKJOT3aMSbmcJnMlEu-_Vg","elasticsearch.node.id":"JNoURkKaRC-n-WbKLRRb1A","elasticsearch.node.name":"Server","elasticsearch.cluster.name":"companyElasticCluster"}
{"@timestamp":"2024-04-23T22:58:00.026Z", "log.level": "INFO", "message":"Deleting expired data", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[Server][generic][T#3]","log.logger":"org.elasticsearch.xpack.ml.action.TransportDeleteExpiredDataAction","elasticsearch.cluster.uuid":"TKJOT3aMSbmcJnMlEu-_Vg","elasticsearch.node.id":"JNoURkKaRC-n-WbKLRRb1A","elasticsearch.node.name":"Server","elasticsearch.cluster.name":"companyElasticCluster"}
{"@timestamp":"2024-04-23T22:58:00.102Z", "log.level": "INFO", "message":"Successfully deleted [0] unused stats documents", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[Server][generic][T#1]","log.logger":"org.elasticsearch.xpack.ml.job.retention.UnusedStatsRemover","elasticsearch.cluster.uuid":"TKJOT3aMSbmcJnMlEu-_Vg","elasticsearch.node.id":"JNoURkKaRC-n-WbKLRRb1A","elasticsearch.node.name":"Server","elasticsearch.cluster.name":"companyElasticCluster"}
{"@timestamp":"2024-04-23T22:58:00.102Z", "log.level": "INFO", "message":"Completed deletion of expired ML data", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[Server][ml_utility][T#2]","log.logger":"org.elasticsearch.xpack.ml.action.TransportDeleteExpiredDataAction","elasticsearch.cluster.uuid":"TKJOT3aMSbmcJnMlEu-_Vg","elasticsearch.node.id":"JNoURkKaRC-n-WbKLRRb1A","elasticsearch.node.name":"Server","elasticsearch.cluster.name":"companyElasticCluster"}
{"@timestamp":"2024-04-23T22:58:00.103Z", "log.level": "INFO", "message":"Successfully completed [ML] maintenance task: triggerDeleteExpiredDataTask", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[Server][ml_utility][T#2]","log.logger":"org.elasticsearch.xpack.ml.MlDailyMaintenanceService","elasticsearch.cluster.uuid":"TKJOT3aMSbmcJnMlEu-_Vg","elasticsearch.node.id":"JNoURkKaRC-n-WbKLRRb1A","elasticsearch.node.name":"Server","elasticsearch.cluster.name":"companyElasticCluster"}
{"@timestamp":"2024-04-24T01:30:00.003Z", "log.level": "INFO", "message":"starting SLM retention snapshot cleanup task", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[Server][trigger_engine_scheduler][T#1]","log.logger":"org.elasticsearch.xpack.slm.SnapshotRetentionTask","elasticsearch.cluster.uuid":"TKJOT3aMSbmcJnMlEu-_Vg","elasticsearch.node.id":"JNoURkKaRC-n-WbKLRRb1A","elasticsearch.node.name":"Server","elasticsearch.cluster.name":"companyElasticCluster"}
{"@timestamp":"2024-04-24T01:30:00.013Z", "log.level": "INFO", "message":"there are no repositories to fetch, SLM retention snapshot cleanup task complete", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[Server][trigger_engine_scheduler][T#1]","log.logger":"org.elasticsearch.xpack.slm.SnapshotRetentionTask","elasticsearch.cluster.uuid":"TKJOT3aMSbmcJnMlEu-_Vg","elasticsearch.node.id":"JNoURkKaRC-n-WbKLRRb1A","elasticsearch.node.name":"Server","elasticsearch.cluster.name":"companyElasticCluster"}
{"@timestamp":"2024-04-24T03:55:14.315Z", "log.level": "INFO", "message":"[.ds-auditbeat-8.13.2-2024.04.23-000001/JFUCbNguQ9S9EVWdharnEA] update_mapping [_doc]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[Server][masterService#updateTask][T#143]","log.logger":"org.elasticsearch.cluster.metadata.MetadataMappingService","elasticsearch.cluster.uuid":"TKJOT3aMSbmcJnMlEu-_Vg","elasticsearch.node.id":"JNoURkKaRC-n-WbKLRRb1A","elasticsearch.node.name":"Server","elasticsearch.cluster.name":"companyElasticCluster"}
{"@timestamp":"2024-04-24T04:12:03.119Z", "log.level": "WARN", "message":"this node is locked into cluster UUID [TKJOT3aMSbmcJnMlEu-_Vg] but [cluster.initial_master_nodes] is set to [Server]; remove this setting to avoid possible data loss caused by subsequent cluster bootstrap attempts; for further information see https://www.elastic.co/guide/en/elasticsearch/reference/8.13/important-settings.html#initial_master_nodes", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[Server][scheduler][T#1]","log.logger":"org.elasticsearch.cluster.coordination.ClusterBootstrapService","elasticsearch.cluster.uuid":"TKJOT3aMSbmcJnMlEu-_Vg","elasticsearch.node.id":"JNoURkKaRC-n-WbKLRRb1A","elasticsearch.node.name":"Server","elasticsearch.cluster.name":"companyElasticCluster"}


[2024-04-24T00:00:10,297][INFO ][o.e.c.m.MetadataMappingService] [Server] [.ds-packetbeat-8.13.2-2024.04.23-000001/dEK4VvXGRR29KRwh3zKYIg] update_mapping [_doc]
[2024-04-24T00:58:00,001][INFO ][o.e.x.m.MlDailyMaintenanceService] [Server] triggering scheduled [ML] maintenance tasks
[2024-04-24T00:58:00,026][INFO ][o.e.x.m.a.TransportDeleteExpiredDataAction] [Server] Deleting expired data
[2024-04-24T00:58:00,102][INFO ][o.e.x.m.j.r.UnusedStatsRemover] [Server] Successfully deleted [0] unused stats documents
[2024-04-24T00:58:00,102][INFO ][o.e.x.m.a.TransportDeleteExpiredDataAction] [Server] Completed deletion of expired ML data
[2024-04-24T00:58:00,103][INFO ][o.e.x.m.MlDailyMaintenanceService] [Server] Successfully completed [ML] maintenance task: triggerDeleteExpiredDataTask
[2024-04-24T03:30:00,003][INFO ][o.e.x.s.SnapshotRetentionTask] [Server] starting SLM retention snapshot cleanup task
[2024-04-24T03:30:00,013][INFO ][o.e.x.s.SnapshotRetentionTask] [Server] there are no repositories to fetch, SLM retention snapshot cleanup task complete
[2024-04-24T05:55:14,315][INFO ][o.e.c.m.MetadataMappingService] [Server] [.ds-auditbeat-8.13.2-2024.04.23-000001/JFUCbNguQ9S9EVWdharnEA] update_mapping [_doc]
[2024-04-24T06:12:03,119][WARN ][o.e.c.c.ClusterBootstrapService] [Server] this node is locked into cluster UUID [TKJOT3aMSbmcJnMlEu-_Vg] but [cluster.initial_master_nodes] is set to [Server]; remove this setting to avoid possible data loss caused by subsequent cluster bootstrap attempts; for further information see https://www.elastic.co/guide/en/elasticsearch/reference/8.13/important-settings.html#initial_master_nodes

Do you need any other log file?

image

4

Could you share the logs from the start of the service? I guess it's what we can see in ElasticCluster_2024_04_23-1.log.gz.

5

I sort of fixed it but Im still getting curl: (52) Empty reply from server when trying to do a curl "health pretty"

What logs do you need?

6

What is the exact request?
What are the full logs from the restart?

7

Most simple request:

curl -X GET "http://localhost:9200/_cluster/health?pretty"

The logs:

[2024-04-30T09:08:09,054][INFO ][o.e.n.Node               ] [server] stopping ...
[2024-04-30T09:08:09,057][INFO ][o.e.c.f.AbstractFileWatchingService] [server] shutting down watcher thread
[2024-04-30T09:08:09,060][INFO ][o.e.c.f.AbstractFileWatchingService] [server] watcher service stopped
[2024-04-30T09:08:09,061][INFO ][o.e.x.w.WatcherService   ] [server] stopping watch service, reason [shutdown initiated]
[2024-04-30T09:08:09,061][INFO ][o.e.x.w.WatcherLifeCycleService] [server] watcher has stopped and shutdown
[2024-04-30T09:08:09,065][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [server] [controller/1843] [Main.cc@176] ML controller exiting
[2024-04-30T09:08:09,066][INFO ][o.e.x.m.p.NativeController] [server] Native controller process has stopped - no new native processes can be started
[2024-04-30T09:08:12,851][INFO ][o.e.n.Node               ] [server] stopped
[2024-04-30T09:08:12,851][INFO ][o.e.n.Node               ] [server] closing ...
[2024-04-30T09:08:12,889][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [server] evicted [0] entries from cache after reloading database [/tmp/elasticsearch-12221783757989146854/geoip-databases/MOFXJpsyRSi1h-vco7V2Qw/GeoLite2-Country.mmdb]
[2024-04-30T09:08:12,890][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [server] evicted [500] entries from cache after reloading database [/tmp/elasticsearch-12221783757989146854/geoip-databases/MOFXJpsyRSi1h-vco7V2Qw/GeoLite2-ASN.mmdb]
[2024-04-30T09:08:12,890][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [server] evicted [500] entries from cache after reloading database [/tmp/elasticsearch-12221783757989146854/geoip-databases/MOFXJpsyRSi1h-vco7V2Qw/GeoLite2-City.mmdb]
[2024-04-30T09:08:12,929][INFO ][o.e.n.Node               ] [server] closed
[2024-04-30T09:08:21,342][INFO ][o.e.n.NativeAccess       ] [server] Using [jdk] native provider and native methods for [Linux]
[2024-04-30T09:08:21,805][INFO ][o.a.l.i.v.PanamaVectorizationProvider] [server] Java vector incubator API enabled; uses preferredBitSize=256; FMA enabled
[2024-04-30T09:08:22,655][INFO ][o.e.n.Node               ] [server] version[8.13.2], pid[2997142], build[rpm/16cc90cd2d08a3147ce02b07e50894bc060a4cbf/2024-04-05T14:45:26.420424304Z], OS[Linux/5.14.0-437.el9.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/21.0.2/21.0.2+13-58]
[2024-04-30T09:08:22,656][INFO ][o.e.n.Node               ] [server] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2024-04-30T09:08:22,656][INFO ][o.e.n.Node               ] [server] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=org.elasticsearch.preallocate, --enable-native-access=org.elasticsearch.nativeaccess, -XX:ReplayDataFile=/var/log/elasticsearch/replay_pid%p.log, -Des.distribution.type=rpm, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-11523029729499202865, --add-modules=jdk.incubator.vector, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,level,pid,tags:filecount=32,filesize=64m, -Xms15920m, -Xmx15920m, -XX:MaxDirectMemorySize=8346664960, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=25, --module-path=/usr/share/elasticsearch/lib, --add-modules=jdk.net, --add-modules=ALL-MODULE-PATH, -Djdk.module.main=org.elasticsearch.server]
[2024-04-30T09:08:22,657][INFO ][o.e.n.Node               ] [server] Default Locale [en_US]
[2024-04-30T09:08:26,890][INFO ][o.e.p.PluginsService     ] [server] loaded module [repository-url]
[2024-04-30T09:08:26,891][INFO ][o.e.p.PluginsService     ] [server] loaded module [rest-root]
[2024-04-30T09:08:26,891][INFO ][o.e.p.PluginsService     ] [server] loaded module [x-pack-core]
[2024-04-30T09:08:26,891][INFO ][o.e.p.PluginsService     ] [server] loaded module [x-pack-redact]
[2024-04-30T09:08:26,891][INFO ][o.e.p.PluginsService     ] [server] loaded module [ingest-user-agent]
[2024-04-30T09:08:26,891][INFO ][o.e.p.PluginsService     ] [server] loaded module [x-pack-async-search]
[2024-04-30T09:08:26,891][INFO ][o.e.p.PluginsService     ] [server] loaded module [x-pack-monitoring]
[2024-04-30T09:08:26,891][INFO ][o.e.p.PluginsService     ] [server] loaded module [repository-s3]
[2024-04-30T09:08:26,891][INFO ][o.e.p.PluginsService     ] [server] loaded module [x-pack-analytics]
[2024-04-30T09:08:26,891][INFO ][o.e.p.PluginsService     ] [server] loaded module [x-pack-ent-search]
[2024-04-30T09:08:26,892][INFO ][o.e.p.PluginsService     ] [server] loaded module [x-pack-autoscaling]
[2024-04-30T09:08:26,892][INFO ][o.e.p.PluginsService     ] [server] loaded module [lang-painless]
[2024-04-30T09:08:26,892][INFO ][o.e.p.PluginsService     ] [server] loaded module [x-pack-ml]
[2024-04-30T09:08:26,892][INFO ][o.e.p.PluginsService     ] [server] loaded module [lang-mustache]
[2024-04-30T09:08:26,892][INFO ][o.e.p.PluginsService     ] [server] loaded module [legacy-geo]
[2024-04-30T09:08:26,892][INFO ][o.e.p.PluginsService     ] [server] loaded module [x-pack-ql]
[2024-04-30T09:08:26,892][INFO ][o.e.p.PluginsService     ] [server] loaded module [rank-rrf]
[2024-04-30T09:08:26,892][INFO ][o.e.p.PluginsService     ] [server] loaded module [analysis-common]
[2024-04-30T09:08:26,892][INFO ][o.e.p.PluginsService     ] [server] loaded module [health-shards-availability]
[2024-04-30T09:08:26,893][INFO ][o.e.p.PluginsService     ] [server] loaded module [transport-netty4]
[2024-04-30T09:08:26,893][INFO ][o.e.p.PluginsService     ] [server] loaded module [aggregations]
[2024-04-30T09:08:26,900][INFO ][o.e.p.PluginsService     ] [server] loaded module [ingest-common]
[2024-04-30T09:08:26,901][INFO ][o.e.p.PluginsService     ] [server] loaded module [frozen-indices]
[2024-04-30T09:08:26,901][INFO ][o.e.p.PluginsService     ] [server] loaded module [x-pack-identity-provider]
[2024-04-30T09:08:26,901][INFO ][o.e.p.PluginsService     ] [server] loaded module [x-pack-text-structure]
[2024-04-30T09:08:26,901][INFO ][o.e.p.PluginsService     ] [server] loaded module [x-pack-shutdown]
[2024-04-30T09:08:26,901][INFO ][o.e.p.PluginsService     ] [server] loaded module [snapshot-repo-test-kit]
[2024-04-30T09:08:26,901][INFO ][o.e.p.PluginsService     ] [server] loaded module [ml-package-loader]
[2024-04-30T09:08:26,901][INFO ][o.e.p.PluginsService     ] [server] loaded module [kibana]
[2024-04-30T09:08:26,902][INFO ][o.e.p.PluginsService     ] [server] loaded module [constant-keyword]
[2024-04-30T09:08:26,902][INFO ][o.e.p.PluginsService     ] [server] loaded module [x-pack-logstash]
[2024-04-30T09:08:26,902][INFO ][o.e.p.PluginsService     ] [server] loaded module [x-pack-graph]
[2024-04-30T09:08:26,902][INFO ][o.e.p.PluginsService     ] [server] loaded module [x-pack-ccr]
[2024-04-30T09:08:26,902][INFO ][o.e.p.PluginsService     ] [server] loaded module [x-pack-esql]
[2024-04-30T09:08:26,902][INFO ][o.e.p.PluginsService     ] [server] loaded module [parent-join]
[2024-04-30T09:08:26,902][INFO ][o.e.p.PluginsService     ] [server] loaded module [counted-keyword]
[2024-04-30T09:08:26,902][INFO ][o.e.p.PluginsService     ] [server] loaded module [x-pack-enrich]
[2024-04-30T09:08:26,902][INFO ][o.e.p.PluginsService     ] [server] loaded module [repositories-metering-api]
[2024-04-30T09:08:26,903][INFO ][o.e.p.PluginsService     ] [server] loaded module [transform]
[2024-04-30T09:08:26,903][INFO ][o.e.p.PluginsService     ] [server] loaded module [repository-azure]
[2024-04-30T09:08:26,903][INFO ][o.e.p.PluginsService     ] [server] loaded module [repository-gcs]
[2024-04-30T09:08:26,903][INFO ][o.e.p.PluginsService     ] [server] loaded module [spatial]
[2024-04-30T09:08:26,903][INFO ][o.e.p.PluginsService     ] [server] loaded module [mapper-version]
[2024-04-30T09:08:26,906][INFO ][o.e.p.PluginsService     ] [server] loaded module [apm]
[2024-04-30T09:08:26,906][INFO ][o.e.p.PluginsService     ] [server] loaded module [mapper-extras]
[2024-04-30T09:08:26,906][INFO ][o.e.p.PluginsService     ] [server] loaded module [x-pack-rollup]
[2024-04-30T09:08:26,906][INFO ][o.e.p.PluginsService     ] [server] loaded module [percolator]
[2024-04-30T09:08:26,906][INFO ][o.e.p.PluginsService     ] [server] loaded module [data-streams]
[2024-04-30T09:08:26,906][INFO ][o.e.p.PluginsService     ] [server] loaded module [x-pack-stack]
[2024-04-30T09:08:26,907][INFO ][o.e.p.PluginsService     ] [server] loaded module [reindex]
[2024-04-30T09:08:26,909][INFO ][o.e.p.PluginsService     ] [server] loaded module [rank-eval]
[2024-04-30T09:08:26,909][INFO ][o.e.p.PluginsService     ] [server] loaded module [systemd]
[2024-04-30T09:08:26,909][INFO ][o.e.p.PluginsService     ] [server] loaded module [x-pack-security]
[2024-04-30T09:08:26,909][INFO ][o.e.p.PluginsService     ] [server] loaded module [blob-cache]
[2024-04-30T09:08:26,910][INFO ][o.e.p.PluginsService     ] [server] loaded module [searchable-snapshots]
[2024-04-30T09:08:26,910][INFO ][o.e.p.PluginsService     ] [server] loaded module [x-pack-slm]
[2024-04-30T09:08:26,910][INFO ][o.e.p.PluginsService     ] [server] loaded module [snapshot-based-recoveries]
[2024-04-30T09:08:26,910][INFO ][o.e.p.PluginsService     ] [server] loaded module [x-pack-watcher]
[2024-04-30T09:08:26,910][INFO ][o.e.p.PluginsService     ] [server] loaded module [old-lucene-versions]
[2024-04-30T09:08:26,910][INFO ][o.e.p.PluginsService     ] [server] loaded module [x-pack-ilm]
[2024-04-30T09:08:26,910][INFO ][o.e.p.PluginsService     ] [server] loaded module [x-pack-voting-only-node]
[2024-04-30T09:08:26,910][INFO ][o.e.p.PluginsService     ] [server] loaded module [x-pack-inference]
[2024-04-30T09:08:26,910][INFO ][o.e.p.PluginsService     ] [server] loaded module [x-pack-deprecation]
[2024-04-30T09:08:26,910][INFO ][o.e.p.PluginsService     ] [server] loaded module [x-pack-fleet]
[2024-04-30T09:08:26,910][INFO ][o.e.p.PluginsService     ] [server] loaded module [x-pack-aggregate-metric]
[2024-04-30T09:08:26,910][INFO ][o.e.p.PluginsService     ] [server] loaded module [x-pack-downsample]
[2024-04-30T09:08:26,911][INFO ][o.e.p.PluginsService     ] [server] loaded module [x-pack-profiling]
[2024-04-30T09:08:26,911][INFO ][o.e.p.PluginsService     ] [server] loaded module [ingest-geoip]
[2024-04-30T09:08:26,911][INFO ][o.e.p.PluginsService     ] [server] loaded module [x-pack-write-load-forecaster]
[2024-04-30T09:08:26,911][INFO ][o.e.p.PluginsService     ] [server] loaded module [search-business-rules]
[2024-04-30T09:08:26,911][INFO ][o.e.p.PluginsService     ] [server] loaded module [ingest-attachment]
[2024-04-30T09:08:26,911][INFO ][o.e.p.PluginsService     ] [server] loaded module [wildcard]
[2024-04-30T09:08:26,911][INFO ][o.e.p.PluginsService     ] [server] loaded module [x-pack-apm-data]
[2024-04-30T09:08:26,911][INFO ][o.e.p.PluginsService     ] [server] loaded module [x-pack-sql]
[2024-04-30T09:08:26,911][INFO ][o.e.p.PluginsService     ] [server] loaded module [unsigned-long]
[2024-04-30T09:08:26,911][INFO ][o.e.p.PluginsService     ] [server] loaded module [runtime-fields-common]
[2024-04-30T09:08:26,911][INFO ][o.e.p.PluginsService     ] [server] loaded module [x-pack-async]
[2024-04-30T09:08:26,912][INFO ][o.e.p.PluginsService     ] [server] loaded module [vector-tile]
[2024-04-30T09:08:26,912][INFO ][o.e.p.PluginsService     ] [server] loaded module [lang-expression]
[2024-04-30T09:08:26,912][INFO ][o.e.p.PluginsService     ] [server] loaded module [x-pack-eql]
[2024-04-30T09:08:28,140][INFO ][o.e.e.NodeEnvironment    ] [server] using [1] data paths, mounts [[/data (172.20.5.101:/data)]], net usable_space [554gb], net total_space [589.5gb], types [nfs4]
[2024-04-30T09:08:28,141][INFO ][o.e.e.NodeEnvironment    ] [server] heap size [15.5gb], compressed ordinary object pointers [true]
[2024-04-30T09:08:30,509][INFO ][o.e.n.Node               ] [server] node name [server], node ID [MOFXJpsyRSi1h-vco7V2Qw], cluster name [companycluster], roles [data_frozen, ingest, data_cold, data, remote_cluster_client, master, data_warm, data_content, transform, data_hot, ml]
[2024-04-30T09:08:36,237][INFO ][o.e.f.FeatureService     ] [server] Registered local node features [data_stream.rollover.lazy, desired_node.version_deprecated, features_supported, health.dsl.info, health.extended_repository_indicator, usage.data_tiers.precalculate_stats]
[2024-04-30T09:08:36,746][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [server] [controller/2997220] [Main.cc@123] controller (64 bit): Version 8.13.2 (Build fdd7177d8c1325) Copyright (c) 2024 Elasticsearch BV
[2024-04-30T09:08:37,022][INFO ][o.e.t.a.APM              ] [server] Sending apm metrics is disabled
[2024-04-30T09:08:37,022][INFO ][o.e.t.a.APM              ] [server] Sending apm tracing is disabled
[2024-04-30T09:08:37,047][INFO ][o.e.x.s.Security         ] [server] Security is enabled
[2024-04-30T09:08:37,339][INFO ][o.e.x.s.a.s.FileRolesStore] [server] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2024-04-30T09:08:37,775][INFO ][o.e.x.s.InitialNodeSecurityAutoConfiguration] [server] Auto-configuration will not generate a password for the elastic built-in superuser, as we cannot  determine if there is a terminal attached to the elasticsearch process. You can use the `bin/elasticsearch-reset-password` tool to set the password for the elastic user.
[2024-04-30T09:08:38,078][INFO ][o.e.x.w.Watcher          ] [server] Watcher initialized components at 2024-04-30T07:08:38.078Z
[2024-04-30T09:08:38,137][INFO ][o.e.x.p.ProfilingPlugin  ] [server] Profiling is enabled
[2024-04-30T09:08:38,171][INFO ][o.e.x.p.ProfilingPlugin  ] [server] profiling index templates will not be installed or reinstalled
[2024-04-30T09:08:38,175][INFO ][o.e.x.a.APMPlugin        ] [server] APM ingest plugin is disabled
[2024-04-30T09:08:38,623][INFO ][o.e.t.n.NettyAllocator   ] [server] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=8mb}]
[2024-04-30T09:08:38,648][INFO ][o.e.i.r.RecoverySettings ] [server] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]
[2024-04-30T09:08:38,689][INFO ][o.e.d.DiscoveryModule    ] [server] using discovery type [multi-node] and seed hosts providers [settings]
[2024-04-30T09:08:40,686][INFO ][o.e.n.Node               ] [server] initialized
[2024-04-30T09:08:40,687][INFO ][o.e.n.Node               ] [server] starting ...
[2024-04-30T09:08:40,793][INFO ][o.e.x.s.c.f.PersistentCache] [server] persistent cache index loaded
[2024-04-30T09:08:40,803][INFO ][o.e.x.d.l.DeprecationIndexingComponent] [server] deprecation component started
[2024-04-30T09:08:40,911][INFO ][o.e.t.TransportService   ] [server] publish_address {172.20.5.100:9300}, bound_addresses {0.0.0.0:9300}
[2024-04-30T09:08:43,256][INFO ][o.e.b.BootstrapChecks    ] [server] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2024-04-30T09:08:43,275][WARN ][o.e.c.c.ClusterBootstrapService] [server] this node is locked into cluster UUID [H2A9Q-BGTmuHlZBMYT9VFQ] but [cluster.initial_master_nodes] is set to [server]; remove this setting to avoid possible data loss caused by subsequent cluster bootstrap attempts; for further information see https://www.elastic.co/guide/en/elasticsearch/reference/8.13/important-settings.html#initial_master_nodes
[2024-04-30T09:08:43,473][INFO ][o.e.c.s.MasterService    ] [server] elected-as-master ([1] nodes joined in term 7)[_FINISH_ELECTION_, {server}{MOFXJpsyRSi1h-vco7V2Qw}{KBDzbiypQriVhlgKwWJwGw}{server}{172.20.5.100}{172.20.5.100:9300}{cdfhilmrstw}{8.13.2}{7000099-8503000} completing election], term: 7, version: 2195, delta: master node changed {previous [], current [{server}{MOFXJpsyRSi1h-vco7V2Qw}{KBDzbiypQriVhlgKwWJwGw}{server}{172.20.5.100}{172.20.5.100:9300}{cdfhilmrstw}{8.13.2}{7000099-8503000}]}
[2024-04-30T09:08:43,837][INFO ][o.e.c.s.ClusterApplierService] [server] master node changed {previous [], current [{server}{MOFXJpsyRSi1h-vco7V2Qw}{KBDzbiypQriVhlgKwWJwGw}{server}{172.20.5.100}{172.20.5.100:9300}{cdfhilmrstw}{8.13.2}{7000099-8503000}]}, term: 7, version: 2195, reason: Publication{term=7, version=2195}
[2024-04-30T09:08:43,914][INFO ][o.e.c.f.AbstractFileWatchingService] [server] starting file watcher ...
[2024-04-30T09:08:43,928][INFO ][o.e.c.f.AbstractFileWatchingService] [server] file settings service up and running [tid=57]
[2024-04-30T09:08:43,928][INFO ][o.e.c.c.NodeJoinExecutor ] [server] node-join: [{server}{MOFXJpsyRSi1h-vco7V2Qw}{KBDzbiypQriVhlgKwWJwGw}{server}{172.20.5.100}{172.20.5.100:9300}{cdfhilmrstw}{8.13.2}{7000099-8503000}] with reason [completing election]
[2024-04-30T09:08:43,932][INFO ][o.e.h.AbstractHttpServerTransport] [server] publish_address {172.20.5.100:9200}, bound_addresses {0.0.0.0:9200}
[2024-04-30T09:08:43,962][INFO ][o.e.n.Node               ] [server] started {server}{MOFXJpsyRSi1h-vco7V2Qw}{KBDzbiypQriVhlgKwWJwGw}{server}{172.20.5.100}{172.20.5.100:9300}{cdfhilmrstw}{8.13.2}{7000099-8503000}{ml.allocated_processors_double=4.0, ml.allocated_processors=4, ml.machine_memory=33387298816, transform.config_version=10.0.0, xpack.installed=true, ml.config_version=12.0.0, ml.max_jvm_size=16693329920}
[2024-04-30T09:08:43,989][INFO ][o.e.c.s.ClusterSettings  ] [server] updating [xpack.monitoring.collection.enabled] from [false] to [true]
[2024-04-30T09:08:45,378][INFO ][o.e.x.s.a.RealmsAuthenticator] [server] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2024-04-30T09:08:45,374][INFO ][o.e.x.s.a.RealmsAuthenticator] [server] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2024-04-30T09:08:46,596][INFO ][o.e.x.s.a.RealmsAuthenticator] [server] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2024-04-30T09:08:46,596][INFO ][o.e.x.s.a.RealmsAuthenticator] [server] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2024-04-30T09:08:47,055][ERROR][o.e.x.s.a.e.ReservedRealm] [server] failed to retrieve password hash for reserved user [elastic]
org.elasticsearch.action.UnavailableShardsException: at least one primary shard for the index [.security-7] is unavailable
	at org.elasticsearch.xpack.security.support.SecurityIndexManager.getUnavailableReason(SecurityIndexManager.java:178) ~[?:?]
	at org.elasticsearch.xpack.security.authc.esnative.NativeUsersStore.getReservedUserInfo(NativeUsersStore.java:640) ~[?:?]
	at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.getUserInfo(ReservedRealm.java:272) ~[?:?]
	at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.doAuthenticate(ReservedRealm.java:136) ~[?:?]
	at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:200) ~[?:?]
	at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:105) ~[?:?]
	at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$4(RealmsAuthenticator.java:170) ~[?:?]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117) ~[?:?]
	at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.consumeToken(RealmsAuthenticator.java:263) ~[?:?]
	at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.authenticate(RealmsAuthenticator.java:106) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:167) ~[?:?]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:146) ~[?:?]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:146) ~[?:?]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:146) ~[?:?]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.doAuthenticate(AuthenticatorChain.java:125) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticate(AuthenticatorChain.java:95) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:264) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:152) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:130) ~[?:?]
	at org.elasticsearch.xpack.security.Security.lambda$getHttpTransports$35(Security.java:1758) ~[?:?]
	at org.elasticsearch.xpack.security.Security.lambda$getHttpServerTransportWithHeadersValidator$38(Security.java:1815) ~[?:?]
	at org.elasticsearch.http.netty4.internal.HttpHeadersAuthenticatorUtils.lambda$getValidatorInboundHandler$2(HttpHeadersAuthenticatorUtils.java:47) ~[?:?]
	at org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.requestStart(Netty4HttpHeaderValidator.java:126) ~[?:?]
	at org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.channelRead(Netty4HttpHeaderValidator.java:61) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[?:?]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1383) ~[?:?]
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246) ~[?:?]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295) ~[?:?]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[?:?]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[?:?]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:1583) ~[?:?]
[2024-04-30T09:08:47,062][INFO ][o.e.x.s.a.RealmsAuthenticator] [server] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2024-04-30T09:08:47,098][INFO ][o.e.x.s.a.Realms         ] [server] license mode is [basic], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]
[2024-04-30T09:08:47,110][INFO ][o.e.l.ClusterStateLicenseService] [server] license [3575729b-d36a-431e-bb91-0937d6539cf3] mode [basic] - valid
[2024-04-30T09:08:47,118][INFO ][o.e.g.GatewayService     ] [server] recovered [79] indices into cluster_state
[2024-04-30T09:08:47,769][INFO ][o.e.h.n.s.HealthNodeTaskExecutor] [server] Node [{server}{MOFXJpsyRSi1h-vco7V2Qw}] is selected as the current health node.
[2024-04-30T09:08:47,770][ERROR][o.e.i.g.GeoIpDownloader  ] [server] exception during geoip databases update
org.elasticsearch.ElasticsearchException: not all primary shards of [.geoip_databases] index are active
	at org.elasticsearch.ingest.geoip.GeoIpDownloader.updateDatabases(GeoIpDownloader.java:131) ~[?:?]
	at org.elasticsearch.ingest.geoip.GeoIpDownloader.runDownloader(GeoIpDownloader.java:279) ~[?:?]
	at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:160) ~[?:?]
	at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:59) ~[?:?]
	at org.elasticsearch.persistent.NodePersistentTasksExecutor$1.doRun(NodePersistentTasksExecutor.java:34) ~[elasticsearch-8.13.2.jar:?]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984) ~[elasticsearch-8.13.2.jar:?]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.13.2.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1583) ~[?:?]
[2024-04-30T09:08:47,894][ERROR][o.e.x.s.a.e.ReservedRealm] [server] failed to retrieve password hash for reserved user [elastic]
org.elasticsearch.action.UnavailableShardsException: at least one primary shard for the index [.security-7] is unavailable
	at org.elasticsearch.xpack.security.support.SecurityIndexManager.getUnavailableReason(SecurityIndexManager.java:178) ~[?:?]
	at org.elasticsearch.xpack.security.authc.esnative.NativeUsersStore.getReservedUserInfo(NativeUsersStore.java:640) ~[?:?]
	at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.getUserInfo(ReservedRealm.java:272) ~[?:?]
	at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.doAuthenticate(ReservedRealm.java:136) ~[?:?]
	at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:200) ~[?:?]
	at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:105) ~[?:?]
	at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$4(RealmsAuthenticator.java:170) ~[?:?]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117) ~[?:?]
	at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.consumeToken(RealmsAuthenticator.java:263) ~[?:?]
	at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.authenticate(RealmsAuthenticator.java:106) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:167) ~[?:?]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:146) ~[?:?]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:146) ~[?:?]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:146) ~[?:?]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.doAuthenticate(AuthenticatorChain.java:125) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticate(AuthenticatorChain.java:95) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:264) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:152) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:130) ~[?:?]
	at org.elasticsearch.xpack.security.Security.lambda$getHttpTransports$35(Security.java:1758) ~[?:?]
	at org.elasticsearch.xpack.security.Security.lambda$getHttpServerTransportWithHeadersValidator$38(Security.java:1815) ~[?:?]
	at org.elasticsearch.http.netty4.internal.HttpHeadersAuthenticatorUtils.lambda$getValidatorInboundHandler$2(HttpHeadersAuthenticatorUtils.java:47) ~[?:?]
	at org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.requestStart(Netty4HttpHeaderValidator.java:126) ~[?:?]
	at org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.channelRead(Netty4HttpHeaderValidator.java:61) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[?:?]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1383) ~[?:?]
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246) ~[?:?]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295) ~[?:?]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[?:?]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[?:?]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:1583) ~[?:?]
[2024-04-30T09:08:47,897][INFO ][o.e.x.s.a.RealmsAuthenticator] [server] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2024-04-30T09:08:48,538][ERROR][o.e.x.s.a.e.ReservedRealm] [server] failed to retrieve password hash for reserved user [elastic]
org.elasticsearch.action.UnavailableShardsException: at least one primary shard for the index [.security-7] is unavailable
	at org.elasticsearch.xpack.security.support.SecurityIndexManager.getUnavailableReason(SecurityIndexManager.java:178) ~[?:?]
	at org.elasticsearch.xpack.security.authc.esnative.NativeUsersStore.getReservedUserInfo(NativeUsersStore.java:640) ~[?:?]
	at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.getUserInfo(ReservedRealm.java:272) ~[?:?]
	at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.doAuthenticate(ReservedRealm.java:136) ~[?:?]
	at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:200) ~[?:?]
	at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:105) ~[?:?]
	at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$4(RealmsAuthenticator.java:170) ~[?:?]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117) ~[?:?]
	at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.consumeToken(RealmsAuthenticator.java:263) ~[?:?]
	at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.authenticate(RealmsAuthenticator.java:106) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:167) ~[?:?]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:146) ~[?:?]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:146) ~[?:?]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:146) ~[?:?]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.doAuthenticate(AuthenticatorChain.java:125) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticate(AuthenticatorChain.java:95) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:264) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:152) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:130) ~[?:?]
	at org.elasticsearch.xpack.security.Security.lambda$getHttpTransports$35(Security.java:1758) ~[?:?]
	at org.elasticsearch.xpack.security.Security.lambda$getHttpServerTransportWithHeadersValidator$38(Security.java:1815) ~[?:?]
	at org.elasticsearch.http.netty4.internal.HttpHeadersAuthenticatorUtils.lambda$getValidatorInboundHandler$2(HttpHeadersAuthenticatorUtils.java:47) ~[?:?]
	at org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.requestStart(Netty4HttpHeaderValidator.java:126) ~[?:?]
	at org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.channelRead(Netty4HttpHeaderValidator.java:61) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[?:?]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1383) ~[?:?]
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246) ~[?:?]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295) ~[?:?]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[?:?]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[?:?]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:1583) ~[?:?]
[2024-04-30T09:08:48,542][INFO ][o.e.x.s.a.RealmsAuthenticator] [server] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2024-04-30T09:08:50,527][INFO ][o.e.i.g.DatabaseNodeService] [server] successfully loaded geoip database file [GeoLite2-Country.mmdb]
[2024-04-30T09:08:50,884][INFO ][o.e.i.g.DatabaseNodeService] [server] successfully loaded geoip database file [GeoLite2-ASN.mmdb]
[2024-04-30T09:08:52,543][INFO ][o.e.i.g.DatabaseNodeService] [server] successfully loaded geoip database file [GeoLite2-City.mmdb]
[2024-04-30T09:08:54,505][WARN ][o.e.h.n.Netty4HttpServerTransport] [server] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:59058}
[2024-04-30T09:09:02,612][INFO ][o.e.c.r.a.AllocationService] [server] current.health="YELLOW" message="Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.kibana-observability-ai-assistant-conversations-000001][0], [.apm-source-map][0]]])." previous.health="RED" reason="shards started [[.kibana-observability-ai-assistant-conversations-000001][0], [.apm-source-map][0]]"
[2024-04-30T09:09:03,495][WARN ][o.e.h.n.Netty4HttpServerTransport] [server] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:36884}
[2024-04-30T09:09:33,698][WARN ][o.e.h.n.Netty4HttpServerTransport] [server] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:40592}
[2024-04-30T09:09:45,251][WARN ][o.e.h.n.Netty4HttpServerTransport] [server] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:52254}
[2024-04-30T09:09:50,613][WARN ][o.e.h.n.Netty4HttpServerTransport] [server] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:52260}
[2024-04-30T09:10:28,822][WARN ][o.e.h.n.Netty4HttpServerTransport] [server] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:59108}
[2024-04-30T09:10:42,834][WARN ][o.e.h.n.Netty4HttpServerTransport] [server] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:42186}
[2024-04-30T09:10:49,729][WARN ][o.e.h.n.Netty4HttpServerTransport] [server] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:59716}
[2024-04-30T09:11:18,576][WARN ][o.e.h.n.Netty4HttpServerTransport] [server] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:55112}
[2024-04-30T09:11:38,137][WARN ][o.e.h.n.Netty4HttpServerTransport] [server] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:58982}
[2024-04-30T09:11:46,895][WARN ][o.e.h.n.Netty4HttpServerTransport] [server] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:56032}
[2024-04-30T09:12:17,274][WARN ][o.e.h.n.Netty4HttpServerTransport] [server] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:48480}
[2024-04-30T09:12:18,501][WARN ][o.e.h.n.Netty4HttpServerTransport] [server] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:48492}
[2024-04-30T09:12:19,885][WARN ][o.e.h.n.Netty4HttpServerTransport] [server] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:48508}
[2024-04-30T09:12:52,404][WARN ][o.e.h.n.Netty4HttpServerTransport] [server] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:58844}
[2024-04-30T09:13:12,560][WARN ][o.e.h.n.Netty4HttpServerTransport] [server] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:57022}
[2024-04-30T09:13:17,952][WARN ][o.e.h.n.Netty4HttpServerTransport] [server] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:38424}
[2024-04-30T09:13:43,949][WARN ][o.e.h.n.Netty4HttpServerTransport] [server] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:40972}
[2024-04-30T09:13:44,100][INFO ][o.e.c.s.IndexScopedSettings] [server] [.ds-metrics-elastic_agent.metricbeat-default-2024.04.24-000001] updating [index.time_series.end_time] from [2024-04-30T07:39:13.000Z] to [2024-04-30T07:48:43.000Z]
[2024-04-30T09:13:44,101][INFO ][o.e.c.s.IndexScopedSettings] [server] [.ds-metrics-system.filesystem-default-2024.04.24-000001] updating [index.time_series.end_time] from [2024-04-30T07:39:13.000Z] to [2024-04-30T07:48:43.000Z]
[2024-04-30T09:13:44,101][INFO ][o.e.c.s.IndexScopedSettings] [server] [.ds-metrics-system.cpu-default-2024.04.24-000001] updating [index.time_series.end_time] from [2024-04-30T07:39:13.000Z] to [2024-04-30T07:48:43.000Z]
[2024-04-30T09:13:44,102][INFO ][o.e.c.s.IndexScopedSettings] [server] [.ds-metrics-elastic_agent.elastic_agent-default-2024.04.24-000001] updating [index.time_series.end_time] from [2024-04-30T07:39:13.000Z] to [2024-04-30T07:48:43.000Z]
[2024-04-30T09:13:44,103][INFO ][o.e.c.s.IndexScopedSettings] [server] [.ds-metrics-elastic_agent.filebeat-default-2024.04.24-000001] updating [index.time_series.end_time] from [2024-04-30T07:39:13.000Z] to [2024-04-30T07:48:43.000Z]
[2024-04-30T09:13:44,103][INFO ][o.e.c.s.IndexScopedSettings] [server] [.ds-metrics-system.process.summary-default-2024.04.24-000001] updating [index.time_series.end_time] from [2024-04-30T07:39:13.000Z] to [2024-04-30T07:48:43.000Z]
[2024-04-30T09:13:44,104][INFO ][o.e.c.s.IndexScopedSettings] [server] [.ds-metrics-system.fsstat-default-2024.04.24-000001] updating [index.time_series.end_time] from [2024-04-30T07:39:13.000Z] to [2024-04-30T07:48:43.000Z]
[2024-04-30T09:13:44,104][INFO ][o.e.c.s.IndexScopedSettings] [server] [.ds-metrics-system.uptime-default-2024.04.24-000001] updating [index.time_series.end_time] from [2024-04-30T07:39:13.000Z] to [2024-04-30T07:48:43.000Z]
[2024-04-30T09:13:44,105][INFO ][o.e.c.s.IndexScopedSettings] [server] [.ds-metrics-system.socket_summary-default-2024.04.24-000001] updating [index.time_series.end_time] from [2024-04-30T07:39:13.000Z] to [2024-04-30T07:48:43.000Z]
[2024-04-30T09:13:44,105][INFO ][o.e.c.s.IndexScopedSettings] [server] [.ds-metrics-system.diskio-default-2024.04.24-000001] updating [index.time_series.end_time] from [2024-04-30T07:39:13.000Z] to [2024-04-30T07:48:43.000Z]
[2024-04-30T09:13:44,106][INFO ][o.e.c.s.IndexScopedSettings] [server] [.ds-metrics-system.memory-default-2024.04.24-000001] updating [index.time_series.end_time] from [2024-04-30T07:39:13.000Z] to [2024-04-30T07:48:43.000Z]
[2024-04-30T09:13:44,106][INFO ][o.e.c.s.IndexScopedSettings] [server] [.ds-metrics-windows.service-default-2024.04.25-000001] updating [index.time_series.end_time] from [2024-04-30T07:39:13.000Z] to [2024-04-30T07:48:43.000Z]
[2024-04-30T09:13:44,106][INFO ][o.e.c.s.IndexScopedSettings] [server] [.ds-metrics-system.load-default-2024.04.24-000001] updating [index.time_series.end_time] from [2024-04-30T07:39:13.000Z] to [2024-04-30T07:48:43.000Z]
[2024-04-30T09:13:44,107][INFO ][o.e.c.s.IndexScopedSettings] [server] [.ds-metrics-system.process-default-2024.04.24-000001] updating [index.time_series.end_time] from [2024-04-30T07:39:13.000Z] to [2024-04-30T07:48:43.000Z]
[2024-04-30T09:13:44,107][INFO ][o.e.c.s.IndexScopedSettings] [server] [.ds-metrics-system.network-default-2024.04.24-000001] updating [index.time_series.end_time] from [2024-04-30T07:39:13.000Z] to [2024-04-30T07:48:43.000Z]
[2024-04-30T09:13:48,729][WARN ][o.e.h.n.Netty4HttpServerTransport] [server] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:40976}
[2024-04-30T09:14:06,050][WARN ][o.e.h.n.Netty4HttpServerTransport] [server] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:48668}

I enter Kibana and logs SEEM to be getting collected but that health check should be working

8

Try with https instead.

9 
curl: (60) SSL certificate problem: self-signed certificate in certificate chain
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

I dont plan to use HTTPS as this will mostly be all internal and generating a cert from our CA and renewing it....zzz

10

Add the -k option

11

This seems to be working:

curl -k -u user:pass -X GET "https://localhost:9200/_cluster/health?pretty"

{
  "cluster_name" : "cluster",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 79,
  "active_shards" : 79,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 29,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 73.14814814814815
}
12

So Elasticsearch is running well and we know that now.

You can look at the settings for Kibana to find out what is the difference...

From the first logs you shared:

"message": "Unable to retrieve version information from Elasticsearch nodes. connect ECONNREFUSED 172.20.5.100:9200"

We can notice a difference with:

https://localhost:9200/_cluster/health?pretty

One is 172.20.5.100, the other localhost.
One suggest http while the other explicitly calls https.
And there a self-signed certificate we bypassed with the -k option. You need to provide the self-signed certificate to Kibana.

But all that is another "story" and you should now open a new discussion in Kibana on how to properly install Kibana on your machine.
But 1st, I'd suggest that you follow the instructions written at Install Kibana from archive on Linux or macOS | Kibana Guide [8.13] | Elastic

13

I just remembered that I have to SSL it for another reason....F my life....Making a new topic because its a bit offtopic