Elasticsearch rolling upgrade failed

rachelyang · August 14, 2024, 1:40pm

I did rolling upgrade for my cluster with two nodes from 8.8 to 8.15. I got 503 error. Here is the result.

curl -X GET -k -u elastic:PASSWORD 'https://localhost:9200'
{"error":{"root_cause":[{"type":"status_exception","reason":"Cluster state has not been recovered yet, cannot write to the [null] index"}],"type":"authentication_processing_error","reason":"failed to promote the auto-configured elastic password hash","caused_by":{"type":"status_exception","reason":"Cluster state has not been recovered yet, cannot write to the [null] index"}},"status":503}

What can I do to recover the cluster?
Thanks

leandrojmp · August 14, 2024, 1:43pm

A two node cluster is not resilient, so it does not support rolling upgrade.

You need to stop both nodes, upgrade them, and start again.

rachelyang · August 14, 2024, 1:47pm

Thanks for your reply. I have done upgrade one by one. They are both the latest version. What can I do to recover cluster now?

leandrojmp · August 14, 2024, 1:56pm

You need to provide more context.

You have two nodes, right? Did you already stopped both of them and upgraded to the same version?

After you started the nodes, what do you have in the logs of each one of them?

Please also share the elasticsearch.yml of both nodes.

rachelyang · August 14, 2024, 2:27pm

I stopped one node, did upgrade, restarted; then repeated to the second node. both to version 8.15
node1:

cluster.name: ElasticSearch
node.name: cvelk5
node.roles: [ master, data,data_content, ingest, remote_cluster_client ]

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
path.repo: ["/mnt/elk/backup"]

bootstrap.memory_lock: true
network.host: 0.0.0.0

http.port: 9200
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/elastic-certificates.p12
  truststore.path: certs/elastic-certificates.p12

xpack.monitoring.exporters.my_local:
  type: local
  use_ingest: false
discovery.seed_hosts: ["cvelk5","cvelk6"]

http.host: 0.0.0.0
transport.host: 0.0.0.0

node2:

cluster.name: ElasticSearch
node.name: cvelk6
node.roles: [ master, data, data_content, ingest, remote_cluster_client ]
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
path.repo: ["/mnt/elk/backup"]

bootstrap.memory_lock: true
network.host: 0.0.0.0
http.port: 9200
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/elastic-certificates.p12
  truststore.path: certs/elastic-certificates.p12
xpack.monitoring.exporters.my_local:
  type: local
  use_ingest: false
discovery.seed_hosts: ["cvelk5", "cvelk6"]
http.host: 0.0.0.0
transport.host: 0.0.0.0

The logs do not have any error, but some warnings.

[cvelk5] An error occurred while attempting to authenticate [elastic] against realm [reserved]
org.elasticsearch.ElasticsearchAuthenticationProcessingError: failed to promote the auto-configured elastic password hash 
Caused by: org.elasticsearch.ElasticsearchStatusException: Cluster state has not been recovered yet, cannot write to the [null] index

[cvelk5] path: /, params: {}, status: 503
org.elasticsearch.ElasticsearchAuthenticationProcessingError: failed to promote the auto-configured elastic password hash
Caused by: org.elasticsearch.ElasticsearchStatusException: Cluster state has not been recovered yet, cannot write to the [null] index

Here is the result of curl on node2:

 curl -X GET -k -u elastic:PASSWORD 'https://localhost:9200'
{"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":["Bearer realm=\"security\"","ApiKey","Basic realm=\"security\", charset=\"UTF-8\""]}}],"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":["Bearer realm=\"security\"","ApiKey","Basic realm=\"security\", charset=\"UTF-8\""]}},"status":401}

leandrojmp · August 14, 2024, 2:34pm

It is unclear what is the issue, you need to share more logs.

Stop both nodes, start them and get the logs since they start, it generates a lot more of logs that could provide more insight.

rachelyang · August 14, 2024, 2:39pm

logs from node1:

[2024-08-14T14:09:33,293][INFO ][o.e.n.NativeAccess       ] [cvelk5] Using native vector library; to disable start with -Dorg.elasticsearch.nativeaccess.enableVectorLibrary=false
[2024-08-14T14:09:33,523][INFO ][o.e.n.NativeAccess       ] [cvelk5] Using [jdk] native provider and native methods for [Linux]
[2024-08-14T14:09:34,604][WARN ][o.a.l.i.v.VectorizationProvider] [cvelk5] Java vector incubator module is not readable. For optimal vector performance, pass --add-modules jdk.incubator.vector to enable Vector API.
[2024-08-14T14:09:35,356][INFO ][o.e.n.Node               ] [cvelk5] version[8.15.0], pid[1305117], build[deb/1a77947f34deddb41af25e6f0ddb8e830159c179/2024-08-05T10:05:34.233336849Z], OS[Linux/5.15.0-75-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/22.0.1/22.0.1+8-16]
[2024-08-14T14:09:35,357][INFO ][o.e.n.Node               ] [cvelk5] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2024-08-14T14:09:35,357][INFO ][o.e.n.Node               ] [cvelk5] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=org.elasticsearch.preallocate, --enable-native-access=org.elasticsearch.nativeaccess,org.apache.lucene.core, -XX:ReplayDataFile=/var/log/elasticsearch/replay_pid%p.log, -Djava.library.path=/usr/share/elasticsearch/lib/platform/linux-x64:/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib, -Djna.library.path=/usr/share/elasticsearch/lib/platform/linux-x64:/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib, -Des.distribution.type=deb, -XX:+UnlockDiagnosticVMOptions, -XX:G1NumCollectionsKeepPinned=10000000, -Xms8g, -Xmx8g, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-9405794003349926722, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,level,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=4294967296, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=25, --module-path=/usr/share/elasticsearch/lib, --add-modules=jdk.net, --add-modules=ALL-MODULE-PATH, -Djdk.module.main=org.elasticsearch.server]
[2024-08-14T14:09:35,358][INFO ][o.e.n.Node               ] [cvelk5] Default Locale [en_US]
[2024-08-14T14:09:38,659][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [repository-url]
[2024-08-14T14:09:38,659][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [rest-root]
[2024-08-14T14:09:38,659][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [x-pack-core]
[2024-08-14T14:09:38,660][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [x-pack-redact]
[2024-08-14T14:09:38,660][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [ingest-user-agent]
[2024-08-14T14:09:38,660][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [x-pack-async-search]
[2024-08-14T14:09:38,660][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [x-pack-monitoring]
[2024-08-14T14:09:38,661][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [repository-s3]
[2024-08-14T14:09:38,661][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [x-pack-analytics]
[2024-08-14T14:09:38,661][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [x-pack-esql-core]
[2024-08-14T14:09:38,662][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [x-pack-ent-search]
[2024-08-14T14:09:38,662][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [x-pack-autoscaling]
[2024-08-14T14:09:38,662][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [lang-painless]
[2024-08-14T14:09:38,662][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [x-pack-ml]
[2024-08-14T14:09:38,662][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [lang-mustache]
[2024-08-14T14:09:38,662][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [legacy-geo]
[2024-08-14T14:09:38,663][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [x-pack-ql]
[2024-08-14T14:09:38,663][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [rank-rrf]
[2024-08-14T14:09:38,663][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [analysis-common]
[2024-08-14T14:09:38,663][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [health-shards-availability]
[2024-08-14T14:09:38,663][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [transport-netty4]
[2024-08-14T14:09:38,663][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [aggregations]
[2024-08-14T14:09:38,664][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [ingest-common]
[2024-08-14T14:09:38,664][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [x-pack-identity-provider]
[2024-08-14T14:09:38,664][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [frozen-indices]
[2024-08-14T14:09:38,664][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [x-pack-text-structure]
[2024-08-14T14:09:38,665][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [x-pack-shutdown]
[2024-08-14T14:09:38,665][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [snapshot-repo-test-kit]
[2024-08-14T14:09:38,665][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [ml-package-loader]
[2024-08-14T14:09:38,665][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [kibana]
[2024-08-14T14:09:38,665][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [constant-keyword]
[2024-08-14T14:09:38,666][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [x-pack-logstash]
[2024-08-14T14:09:38,666][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [x-pack-ccr]
[2024-08-14T14:09:38,666][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [x-pack-graph]
[2024-08-14T14:09:38,666][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [x-pack-esql]
[2024-08-14T14:09:38,666][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [parent-join]
[2024-08-14T14:09:38,667][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [x-pack-enrich]
[2024-08-14T14:09:38,667][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [counted-keyword]
[2024-08-14T14:09:38,667][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [repositories-metering-api]
[2024-08-14T14:09:38,668][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [transform]
[2024-08-14T14:09:38,668][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [repository-azure]
[2024-08-14T14:09:38,668][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [repository-gcs]
[2024-08-14T14:09:38,668][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [spatial]
[2024-08-14T14:09:38,668][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [mapper-version]
[2024-08-14T14:09:38,669][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [mapper-extras]
[2024-08-14T14:09:38,669][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [apm]
[2024-08-14T14:09:38,669][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [x-pack-rollup]
[2024-08-14T14:09:38,669][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [percolator]
[2024-08-14T14:09:38,670][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [x-pack-stack]
[2024-08-14T14:09:38,670][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [data-streams]
[2024-08-14T14:09:38,670][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [systemd]
[2024-08-14T14:09:38,670][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [rank-eval]
[2024-08-14T14:09:38,670][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [reindex]
[2024-08-14T14:09:38,671][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [x-pack-security]
[2024-08-14T14:09:38,671][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [blob-cache]
[2024-08-14T14:09:38,671][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [searchable-snapshots]
[2024-08-14T14:09:38,671][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [x-pack-slm]
[2024-08-14T14:09:38,672][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [x-pack-geoip-enterprise-downloader]
[2024-08-14T14:09:38,672][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [snapshot-based-recoveries]
[2024-08-14T14:09:38,672][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [x-pack-watcher]
[2024-08-14T14:09:38,672][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [old-lucene-versions]
[2024-08-14T14:09:38,673][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [x-pack-ilm]
[2024-08-14T14:09:38,673][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [x-pack-voting-only-node]
[2024-08-14T14:09:38,673][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [x-pack-inference]
[2024-08-14T14:09:38,673][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [x-pack-deprecation]
[2024-08-14T14:09:38,673][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [x-pack-fleet]
[2024-08-14T14:09:38,673][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [x-pack-aggregate-metric]
[2024-08-14T14:09:38,673][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [x-pack-downsample]
[2024-08-14T14:09:38,674][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [x-pack-profiling]
[2024-08-14T14:09:38,674][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [ingest-geoip]
[2024-08-14T14:09:38,674][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [x-pack-write-load-forecaster]
[2024-08-14T14:09:38,674][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [search-business-rules]
[2024-08-14T14:09:38,674][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [ingest-attachment]
[2024-08-14T14:09:38,674][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [wildcard]
[2024-08-14T14:09:38,674][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [x-pack-apm-data]
[2024-08-14T14:09:38,675][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [x-pack-sql]
[2024-08-14T14:09:38,675][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [unsigned-long]
[2024-08-14T14:09:38,675][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [runtime-fields-common]
[2024-08-14T14:09:38,675][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [x-pack-async]
[2024-08-14T14:09:38,675][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [vector-tile]
[2024-08-14T14:09:38,676][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [lang-expression]
[2024-08-14T14:09:38,676][INFO ][o.e.p.PluginsService     ] [cvelk5] loaded module [x-pack-eql]
[2024-08-14T14:09:39,884][INFO ][o.e.e.NodeEnvironment    ] [cvelk5] using [1] data paths, mounts [[/ (/dev/md1p1)]], net usable_space [1.4tb], net total_space [1.7tb], types [ext4]
[2024-08-14T14:09:39,884][INFO ][o.e.e.NodeEnvironment    ] [cvelk5] heap size [8gb], compressed ordinary object pointers [true]
[2024-08-14T14:09:40,014][INFO ][o.e.n.Node               ] [cvelk5] node name [cvelk5], node ID [apRS6ewWRWKWA8_6HsteiQ], cluster name [ElasticSearch], roles [ingest, data_content, master, data, remote_cluster_client]
[2024-08-14T14:09:44,183][INFO ][o.e.i.r.RecoverySettings ] [cvelk5] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]
[2024-08-14T14:09:44,346][INFO ][o.e.f.FeatureService     ] [cvelk5] Registered local node features [data_stream.auto_sharding, data_stream.lifecycle.global_retention, data_stream.rollover.lazy, desired_node.version_deprecated, esql.agg_values, esql.async_query, esql.base64_decode_encode, esql.casting_operator, esql.counter_types, esql.disable_nullable_opts, esql.from_options, esql.metadata_fields, esql.metrics_counter_fields, esql.mv_ordering_sorted_ascending, esql.mv_sort, esql.spatial_points_from_source, esql.spatial_shapes, esql.st_centroid_agg, esql.st_contains_within, esql.st_disjoint, esql.st_intersects, esql.st_x_y, esql.string_literal_auto_casting, esql.string_literal_auto_casting_extended, esql.timespan_abbreviations, features_supported, file_settings, geoip.downloader.database.configuration, health.dsl.info, health.extended_repository_indicator, knn_retriever_supported, license-trial-independent-version, mapper.index_sorting_on_nested, mapper.keyword_dimension_ignore_above, mapper.pass_through_priority, mapper.range.null_values_off_by_one_fix, mapper.source.synthetic_source_fallback, mapper.track_ignored_source, mapper.vectors.bit_vectors, mapper.vectors.int4_quantization, rest.capabilities_action, retrievers_supported, rrf_retriever_supported, script.hamming, search.vectors.k_param_supported, security.migration_framework, security.roles_metadata_flattened, standard_retriever_supported, stats.include_disk_thresholds, text_similarity_reranker_retriever_supported, unified_highlighter_matched_fields, usage.data_tiers.precalculate_stats]
[2024-08-14T14:09:44,774][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [cvelk5] [controller/1305186] [Main.cc@123] controller (64 bit): Version 8.15.0 (Build 64f00009177815) Copyright (c) 2024 Elasticsearch BV
[2024-08-14T14:09:44,988][INFO ][o.e.t.a.APM              ] [cvelk5] Sending apm metrics is disabled
[2024-08-14T14:09:44,988][INFO ][o.e.t.a.APM              ] [cvelk5] Sending apm tracing is disabled
[2024-08-14T14:09:45,024][INFO ][o.e.x.s.Security         ] [cvelk5] Security is enabled
[2024-08-14T14:09:45,320][INFO ][o.e.x.s.a.s.FileRolesStore] [cvelk5] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2024-08-14T14:09:45,489][INFO ][o.e.x.s.InitialNodeSecurityAutoConfiguration] [cvelk5] Auto-configuration will not generate a password for the elastic built-in superuser, as we cannot  determine if there is a terminal attached to the elasticsearch process. You can use the `bin/elasticsearch-reset-password` tool to set the password for the elastic user.
[2024-08-14T14:09:45,714][INFO ][o.e.x.w.Watcher          ] [cvelk5] Watcher initialized components at 2024-08-14T14:09:45.714Z
[2024-08-14T14:09:45,784][INFO ][o.e.x.p.ProfilingPlugin  ] [cvelk5] Profiling is enabled
[2024-08-14T14:09:45,800][INFO ][o.e.x.p.ProfilingPlugin  ] [cvelk5] profiling index templates will not be installed or reinstalled
[2024-08-14T14:09:45,804][INFO ][o.e.x.a.APMPlugin        ] [cvelk5] APM ingest plugin is enabled
[2024-08-14T14:09:45,839][INFO ][o.e.x.a.APMIndexTemplateRegistry] [cvelk5] APM index template registry is enabled
[2024-08-14T14:09:46,318][INFO ][o.e.t.n.NettyAllocator   ] [cvelk5] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2024-08-14T14:09:46,383][INFO ][o.e.d.DiscoveryModule    ] [cvelk5] using discovery type [multi-node] and seed hosts providers [settings]
[2024-08-14T14:09:47,698][INFO ][o.e.n.Node               ] [cvelk5] initialized
[2024-08-14T14:09:47,699][INFO ][o.e.n.Node               ] [cvelk5] starting ...
[2024-08-14T14:09:47,780][INFO ][o.e.x.s.c.f.PersistentCache] [cvelk5] persistent cache index loaded
[2024-08-14T14:09:47,781][INFO ][o.e.x.d.l.DeprecationIndexingComponent] [cvelk5] deprecation component started
[2024-08-14T14:09:47,873][INFO ][o.e.t.TransportService   ] [cvelk5] publish_address {204.141.9.12:9300}, bound_addresses {[::]:9300}
[2024-08-14T14:09:49,117][INFO ][o.e.b.BootstrapChecks    ] [cvelk5] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2024-08-14T14:09:49,122][INFO ][o.e.c.c.ClusterBootstrapService] [cvelk5] this node has not joined a bootstrapped cluster yet; [cluster.initial_master_nodes] is set to []
[2024-08-14T14:09:59,134][WARN ][o.e.c.c.ClusterFormationFailureHelper] [cvelk5] master not discovered yet, this node has not previously joined a bootstrapped cluster, and [cluster.initial_master_nodes] is empty on this node: have discovered [{cvelk5}{apRS6ewWRWKWA8_6HsteiQ}{MIXcSCdWRz-OTivISx5VQQ}{cvelk5}{204.141.9.12}{204.141.9.12:9300}{dimrs}{8.15.0}{7000099-8512000}, {cvelk6}{jkx38DGkQcqlcpxZy6Ih2w}{ASnoIqWTTVunVXkR7IOBWw}{cvelk6}{204.141.9.13}{204.141.9.13:9300}{dimrs}{8.15.0}{7000099-8512000}]; discovery will continue using [127.0.1.1:9300, 204.141.9.13:9300] from hosts providers and [{cvelk5}{apRS6ewWRWKWA8_6HsteiQ}{MIXcSCdWRz-OTivISx5VQQ}{cvelk5}{204.141.9.12}{204.141.9.12:9300}{dimrs}{8.15.0}{7000099-8512000}] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.15/discovery-troubleshooting.html
[2024-08-14T14:10:09,136][WARN ][o.e.c.c.ClusterFormationFailureHelper] [cvelk5] master not discovered yet, this node has not previously joined a bootstrapped cluster, and [cluster.initial_master_nodes] is empty on this node: have discovered [{cvelk5}{apRS6ewWRWKWA8_6HsteiQ}{MIXcSCdWRz-OTivISx5VQQ}{cvelk5}{204.141.9.12}{204.141.9.12:9300}{dimrs}{8.15.0}{7000099-8512000}, {cvelk6}{jkx38DGkQcqlcpxZy6Ih2w}{ASnoIqWTTVunVXkR7IOBWw}{cvelk6}{204.141.9.13}{204.141.9.13:9300}{dimrs}{8.15.0}{7000099-8512000}]; discovery will continue using [127.0.1.1:9300, 204.141.9.13:9300] from hosts providers and [{cvelk5}{apRS6ewWRWKWA8_6HsteiQ}{MIXcSCdWRz-OTivISx5VQQ}{cvelk5}{204.141.9.12}{204.141.9.12:9300}{dimrs}{8.15.0}{7000099-8512000}] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.15/discovery-troubleshooting.html
[2024-08-14T14:10:19,138][WARN ][o.e.c.c.ClusterFormationFailureHelper] [cvelk5] master not discovered yet, this node has not previously joined a bootstrapped cluster, and [cluster.initial_master_nodes] is empty on this node: have discovered [{cvelk5}{apRS6ewWRWKWA8_6HsteiQ}{MIXcSCdWRz-OTivISx5VQQ}{cvelk5}{204.141.9.12}{204.141.9.12:9300}{dimrs}{8.15.0}{7000099-8512000}, {cvelk6}{jkx38DGkQcqlcpxZy6Ih2w}{ASnoIqWTTVunVXkR7IOBWw}{cvelk6}{204.141.9.13}{204.141.9.13:9300}{dimrs}{8.15.0}{7000099-8512000}]; discovery will continue using [127.0.1.1:9300, 204.141.9.13:9300] from hosts providers and [{cvelk5}{apRS6ewWRWKWA8_6HsteiQ}{MIXcSCdWRz-OTivISx5VQQ}{cvelk5}{204.141.9.12}{204.141.9.12:9300}{dimrs}{8.15.0}{7000099-8512000}] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.15/discovery-troubleshooting.html
[2024-08-14T14:10:19,139][WARN ][o.e.n.Node               ] [cvelk5] timed out while waiting for initial discovery state - timeout: 30s
[2024-08-14T14:10:19,149][INFO ][o.e.h.AbstractHttpServerTransport] [cvelk5] publish_address {204.141.9.12:9200}, bound_addresses {[::]:9200}
[2024-08-14T14:10:19,183][INFO ][o.e.n.Node               ] [cvelk5] started {cvelk5}{apRS6ewWRWKWA8_6HsteiQ}{MIXcSCdWRz-OTivISx5VQQ}{cvelk5}{204.141.9.12}{204.141.9.12:9300}{dimrs}{8.15.0}{7000099-8512000}{transform.config_version=10.0.0, ml.config_version=12.0.0, xpack.installed=true}
[2024-08-14T14:10:23,760][WARN ][o.e.x.s.a.RealmsAuthenticator] [cvelk5] An error occurred while attempting to authenticate [elastic] against realm [reserved]
org.elasticsearch.ElasticsearchAuthenticationProcessingError: failed to promote the auto-configured elastic password hash
	at org.elasticsearch.xpack.core.security.support.Exceptions.authenticationProcessError(Exceptions.java:44) ~[?:?]
	at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.lambda$doAuthenticate$2(ReservedRealm.java:160) ~[?:?]
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.15.0.jar:?]
	at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:257) ~[elasticsearch-8.15.0.jar:?]
	at org.elasticsearch.xpack.security.support.SecurityIndexManager.prepareIndexIfNeededThenExecute(SecurityIndexManager.java:591) ~[?:?]
	at org.elasticsearch.xpack.security.authc.esnative.NativeUsersStore.updateReservedUser(NativeUsersStore.java:350) ~[?:?]
	at org.elasticsearch.xpack.security.authc.esnative.NativeUsersStore.createElasticUser(NativeUsersStore.java:336) ~[?:?]
	at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.lambda$doAuthenticate$3(ReservedRealm.java:152) ~[?:?]
	at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.lambda$getUserInfo$7(ReservedRealm.java:274) ~[?:?]
	at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:249) ~[elasticsearch-8.15.0.jar:?]
	at org.elasticsearch.xpack.security.authc.esnative.NativeUsersStore.getReservedUserInfo(NativeUsersStore.java:638) ~[?:?]
	at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.getUserInfo(ReservedRealm.java:272) ~[?:?]
	at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.doAuthenticate(ReservedRealm.java:136) ~[?:?]
	at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:200) ~[?:?]
	at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:105) ~[?:?]
	at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$4(RealmsAuthenticator.java:170) ~[?:?]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117) ~[?:?]
	at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.consumeToken(RealmsAuthenticator.java:263) ~[?:?]
	at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.authenticate(RealmsAuthenticator.java:106) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:167) ~[?:?]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:146) ~[?:?]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:146) ~[?:?]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:146) ~[?:?]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.doAuthenticate(AuthenticatorChain.java:125) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticate(AuthenticatorChain.java:95) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:264) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:152) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:130) ~[?:?]
	at org.elasticsearch.xpack.security.Security.lambda$getHttpTransports$39(Security.java:1974) ~[?:?]
	at org.elasticsearch.xpack.security.Security.lambda$getHttpServerTransportWithHeadersValidator$42(Security.java:2031) ~[?:?]
	at org.elasticsearch.http.netty4.internal.HttpHeadersAuthenticatorUtils.lambda$getValidatorInboundHandler$2(HttpHeadersAuthenticatorUtils.java:47) ~[?:?]
	at org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.lambda$requestStart$1(Netty4HttpHeaderValidator.java:139) ~[?:?]
	at org.elasticsearch.action.ActionListener.run(ActionListener.java:444) ~[elasticsearch-8.15.0.jar:?]
	at org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.requestStart(Netty4HttpHeaderValidator.java:113) ~[?:?]
	at org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.channelRead(Netty4HttpHeaderValidator.java:61) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[?:?]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1475) ~[?:?]
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1338) ~[?:?]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1387) ~[?:?]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530) ~[?:?]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469) ~[?:?]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:1570) ~[?:?]
Caused by: org.elasticsearch.ElasticsearchStatusException: Cluster state has not been recovered yet, cannot write to the [null] index
	at org.elasticsearch.xpack.security.support.SecurityIndexManager.prepareIndexIfNeededThenExecute(SecurityIndexManager.java:492) ~[?:?]
	... 67 more

rachelyang · August 14, 2024, 2:47pm

Logs from node2:

[2024-08-14T13:24:24,681][INFO ][o.e.n.NativeAccess       ] [cvelk6] Using native vector library; to disable start with -Dorg.elasticsearch.nativeaccess.enableVectorLibrary=false
[2024-08-14T13:24:25,035][INFO ][o.e.n.NativeAccess       ] [cvelk6] Using [jdk] native provider and native methods for [Linux]
[2024-08-14T13:24:28,197][INFO ][o.a.l.i.v.PanamaVectorizationProvider] [cvelk6] Java vector incubator API enabled; uses preferredBitSize=256; FMA enabled
[2024-08-14T13:24:28,968][INFO ][o.e.n.Node               ] [cvelk6] version[8.15.0], pid[3230620], build[deb/1a77947f34deddb41af25e6f0ddb8e830159c179/2024-08-05T10:05:34.233336849Z], OS[Linux/5.15.0-75-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/22.0.1/22.0.1+8-16]
[2024-08-14T13:24:28,969][INFO ][o.e.n.Node               ] [cvelk6] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2024-08-14T13:24:28,970][INFO ][o.e.n.Node               ] [cvelk6] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=org.elasticsearch.preallocate, --enable-native-access=org.elasticsearch.nativeaccess,org.apache.lucene.core, -XX:ReplayDataFile=/var/log/elasticsearch/replay_pid%p.log, -Djava.library.path=/usr/share/elasticsearch/lib/platform/linux-x64:/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib, -Djna.library.path=/usr/share/elasticsearch/lib/platform/linux-x64:/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib, -Des.distribution.type=deb, -XX:+UnlockDiagnosticVMOptions, -XX:G1NumCollectionsKeepPinned=10000000, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-9593768476389066128, --add-modules=jdk.incubator.vector, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,level,pid,tags:filecount=32,filesize=64m, -Xms31744m, -Xmx31744m, -XX:MaxDirectMemorySize=16642998272, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=25, --module-path=/usr/share/elasticsearch/lib, --add-modules=jdk.net, --add-modules=ALL-MODULE-PATH, -Djdk.module.main=org.elasticsearch.server]
[2024-08-14T13:24:28,970][INFO ][o.e.n.Node               ] [cvelk6] Default Locale [en_US]
[2024-08-14T13:24:32,149][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [repository-url]
[2024-08-14T13:24:32,150][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [rest-root]
[2024-08-14T13:24:32,150][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [x-pack-core]
[2024-08-14T13:24:32,150][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [x-pack-redact]
[2024-08-14T13:24:32,151][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [ingest-user-agent]
[2024-08-14T13:24:32,151][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [x-pack-async-search]
[2024-08-14T13:24:32,151][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [x-pack-monitoring]
[2024-08-14T13:24:32,151][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [repository-s3]
[2024-08-14T13:24:32,151][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [x-pack-esql-core]
[2024-08-14T13:24:32,152][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [x-pack-analytics]
[2024-08-14T13:24:32,152][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [x-pack-ent-search]
[2024-08-14T13:24:32,152][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [x-pack-autoscaling]
[2024-08-14T13:24:32,152][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [lang-painless]
[2024-08-14T13:24:32,153][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [x-pack-ml]
[2024-08-14T13:24:32,153][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [lang-mustache]
[2024-08-14T13:24:32,153][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [legacy-geo]
[2024-08-14T13:24:32,153][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [x-pack-ql]
[2024-08-14T13:24:32,153][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [rank-rrf]
[2024-08-14T13:24:32,153][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [analysis-common]
[2024-08-14T13:24:32,154][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [health-shards-availability]
[2024-08-14T13:24:32,154][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [transport-netty4]
[2024-08-14T13:24:32,154][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [aggregations]
[2024-08-14T13:24:32,154][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [ingest-common]
[2024-08-14T13:24:32,154][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [x-pack-identity-provider]
[2024-08-14T13:24:32,155][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [frozen-indices]
[2024-08-14T13:24:32,155][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [x-pack-shutdown]
[2024-08-14T13:24:32,155][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [x-pack-text-structure]
[2024-08-14T13:24:32,155][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [snapshot-repo-test-kit]
[2024-08-14T13:24:32,155][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [ml-package-loader]
[2024-08-14T13:24:32,156][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [kibana]
[2024-08-14T13:24:32,156][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [constant-keyword]
[2024-08-14T13:24:32,156][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [x-pack-logstash]
[2024-08-14T13:24:32,156][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [x-pack-graph]
[2024-08-14T13:24:32,156][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [x-pack-ccr]
[2024-08-14T13:24:32,157][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [x-pack-esql]
[2024-08-14T13:24:32,157][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [parent-join]
[2024-08-14T13:24:32,157][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [counted-keyword]
[2024-08-14T13:24:32,157][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [x-pack-enrich]
[2024-08-14T13:24:32,158][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [repositories-metering-api]
[2024-08-14T13:24:32,158][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [transform]
[2024-08-14T13:24:32,158][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [repository-azure]
[2024-08-14T13:24:32,158][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [repository-gcs]
[2024-08-14T13:24:32,158][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [spatial]
[2024-08-14T13:24:32,159][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [mapper-extras]
[2024-08-14T13:24:32,159][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [mapper-version]
[2024-08-14T13:24:32,159][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [apm]
[2024-08-14T13:24:32,159][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [x-pack-rollup]
[2024-08-14T13:24:32,159][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [percolator]
[2024-08-14T13:24:32,159][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [x-pack-stack]
[2024-08-14T13:24:32,160][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [data-streams]
[2024-08-14T13:24:32,160][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [systemd]
[2024-08-14T13:24:32,160][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [reindex]
[2024-08-14T13:24:32,160][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [rank-eval]
[2024-08-14T13:24:32,160][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [x-pack-security]
[2024-08-14T13:24:32,161][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [blob-cache]
[2024-08-14T13:24:32,161][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [searchable-snapshots]
[2024-08-14T13:24:32,161][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [x-pack-slm]
[2024-08-14T13:24:32,161][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [x-pack-geoip-enterprise-downloader]
[2024-08-14T13:24:32,161][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [snapshot-based-recoveries]
[2024-08-14T13:24:32,161][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [x-pack-watcher]
[2024-08-14T13:24:32,161][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [old-lucene-versions]
[2024-08-14T13:24:32,162][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [x-pack-ilm]
[2024-08-14T13:24:32,162][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [x-pack-inference]
[2024-08-14T13:24:32,162][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [x-pack-voting-only-node]
[2024-08-14T13:24:32,162][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [x-pack-deprecation]
[2024-08-14T13:24:32,162][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [x-pack-fleet]
[2024-08-14T13:24:32,163][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [x-pack-aggregate-metric]
[2024-08-14T13:24:32,163][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [x-pack-downsample]
[2024-08-14T13:24:32,163][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [x-pack-profiling]
[2024-08-14T13:24:32,163][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [ingest-geoip]
[2024-08-14T13:24:32,163][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [x-pack-write-load-forecaster]
[2024-08-14T13:24:32,163][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [search-business-rules]
[2024-08-14T13:24:32,163][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [ingest-attachment]
[2024-08-14T13:24:32,164][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [wildcard]
[2024-08-14T13:24:32,164][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [x-pack-apm-data]
[2024-08-14T13:24:32,164][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [x-pack-sql]
[2024-08-14T13:24:32,164][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [unsigned-long]
[2024-08-14T13:24:32,164][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [x-pack-async]
[2024-08-14T13:24:32,164][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [runtime-fields-common]
[2024-08-14T13:24:32,164][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [vector-tile]
[2024-08-14T13:24:32,165][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [lang-expression]
[2024-08-14T13:24:32,165][INFO ][o.e.p.PluginsService     ] [cvelk6] loaded module [x-pack-eql]
[2024-08-14T13:24:33,328][INFO ][o.e.e.NodeEnvironment    ] [cvelk6] using [1] data paths, mounts [[/ (/dev/md1p1)]], net usable_space [1.4tb], net total_space [1.7tb], types [ext4]
[2024-08-14T13:24:33,328][INFO ][o.e.e.NodeEnvironment    ] [cvelk6] heap size [31gb], compressed ordinary object pointers [true]
[2024-08-14T13:24:33,474][INFO ][o.e.n.Node               ] [cvelk6] node name [cvelk6], node ID [jkx38DGkQcqlcpxZy6Ih2w], cluster name [ElasticSearch], roles [data_content, ingest, remote_cluster_client, data, master]
[2024-08-14T13:24:37,688][INFO ][o.e.i.r.RecoverySettings ] [cvelk6] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]
[2024-08-14T13:24:37,848][INFO ][o.e.f.FeatureService     ] [cvelk6] Registered local node features [data_stream.auto_sharding, data_stream.lifecycle.global_retention, data_stream.rollover.lazy, desired_node.version_deprecated, esql.agg_values, esql.async_query, esql.base64_decode_encode, esql.casting_operator, esql.counter_types, esql.disable_nullable_opts, esql.from_options, esql.metadata_fields, esql.metrics_counter_fields, esql.mv_ordering_sorted_ascending, esql.mv_sort, esql.spatial_points_from_source, esql.spatial_shapes, esql.st_centroid_agg, esql.st_contains_within, esql.st_disjoint, esql.st_intersects, esql.st_x_y, esql.string_literal_auto_casting, esql.string_literal_auto_casting_extended, esql.timespan_abbreviations, features_supported, file_settings, geoip.downloader.database.configuration, health.dsl.info, health.extended_repository_indicator, knn_retriever_supported, license-trial-independent-version, mapper.index_sorting_on_nested, mapper.keyword_dimension_ignore_above, mapper.pass_through_priority, mapper.range.null_values_off_by_one_fix, mapper.source.synthetic_source_fallback, mapper.track_ignored_source, mapper.vectors.bit_vectors, mapper.vectors.int4_quantization, rest.capabilities_action, retrievers_supported, rrf_retriever_supported, script.hamming, search.vectors.k_param_supported, security.migration_framework, security.roles_metadata_flattened, standard_retriever_supported, stats.include_disk_thresholds, text_similarity_reranker_retriever_supported, unified_highlighter_matched_fields, usage.data_tiers.precalculate_stats]
[2024-08-14T13:24:38,284][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [cvelk6] [controller/3230705] [Main.cc@123] controller (64 bit): Version 8.15.0 (Build 64f00009177815) Copyright (c) 2024 Elasticsearch BV
[2024-08-14T13:24:38,532][INFO ][o.e.t.a.APM              ] [cvelk6] Sending apm metrics is disabled
[2024-08-14T13:24:38,532][INFO ][o.e.t.a.APM              ] [cvelk6] Sending apm tracing is disabled
[2024-08-14T13:24:38,571][INFO ][o.e.x.s.Security         ] [cvelk6] Security is enabled
[2024-08-14T13:24:38,958][INFO ][o.e.x.s.a.s.FileRolesStore] [cvelk6] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2024-08-14T13:24:39,142][INFO ][o.e.x.s.InitialNodeSecurityAutoConfiguration] [cvelk6] Auto-configuration will not generate a password for the elastic built-in superuser, as we cannot  determine if there is a terminal attached to the elasticsearch process. You can use the `bin/elasticsearch-reset-password` tool to set the password for the elastic user.
[2024-08-14T13:24:39,385][INFO ][o.e.x.w.Watcher          ] [cvelk6] Watcher initialized components at 2024-08-14T13:24:39.384Z
[2024-08-14T13:24:39,453][INFO ][o.e.x.p.ProfilingPlugin  ] [cvelk6] Profiling is enabled
[2024-08-14T13:24:39,471][INFO ][o.e.x.p.ProfilingPlugin  ] [cvelk6] profiling index templates will not be installed or reinstalled
[2024-08-14T13:24:39,475][INFO ][o.e.x.a.APMPlugin        ] [cvelk6] APM ingest plugin is enabled
[2024-08-14T13:24:39,520][INFO ][o.e.x.a.APMIndexTemplateRegistry] [cvelk6] APM index template registry is enabled
[2024-08-14T13:24:39,993][INFO ][o.e.t.n.NettyAllocator   ] [cvelk6] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=16mb}]
[2024-08-14T13:24:40,043][INFO ][o.e.d.DiscoveryModule    ] [cvelk6] using discovery type [multi-node] and seed hosts providers [settings]
[2024-08-14T13:24:41,330][INFO ][o.e.n.Node               ] [cvelk6] initialized
[2024-08-14T13:24:41,331][INFO ][o.e.n.Node               ] [cvelk6] starting ...
[2024-08-14T13:24:41,429][INFO ][o.e.x.s.c.f.PersistentCache] [cvelk6] persistent cache index loaded
[2024-08-14T13:24:41,430][INFO ][o.e.x.d.l.DeprecationIndexingComponent] [cvelk6] deprecation component started
[2024-08-14T13:24:41,522][INFO ][o.e.t.TransportService   ] [cvelk6] publish_address {204.141.9.13:9300}, bound_addresses {[::]:9300}
[2024-08-14T13:24:43,166][INFO ][o.e.b.BootstrapChecks    ] [cvelk6] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2024-08-14T13:24:43,212][INFO ][o.e.c.c.ClusterBootstrapService] [cvelk6] this node is locked into cluster UUID [nTodcM8ER6aQhXegTujWWA] and will not attempt further cluster bootstrapping
[2024-08-14T13:24:53,222][WARN ][o.e.c.c.ClusterFormationFailureHelper] [cvelk6] master not discovered or elected yet, an election requires a node with id [MsUT9oCuSauu5MSea2E_Wg], have only discovered non-quorum [{cvelk6}{jkx38DGkQcqlcpxZy6Ih2w}{ASnoIqWTTVunVXkR7IOBWw}{cvelk6}{204.141.9.13}{204.141.9.13:9300}{dimrs}{8.15.0}{7000099-8512000}, {cvelk5}{apRS6ewWRWKWA8_6HsteiQ}{VktzXDvBRcOzcZrwvTuz9Q}{cvelk5}{204.141.9.12}{204.141.9.12:9300}{dimrs}{8.15.0}{7000099-8512000}]; discovery will continue using [204.141.9.12:9300, 127.0.1.1:9300] from hosts providers and [{cvelk6}{jkx38DGkQcqlcpxZy6Ih2w}{ASnoIqWTTVunVXkR7IOBWw}{cvelk6}{204.141.9.13}{204.141.9.13:9300}{dimrs}{8.15.0}{7000099-8512000}] from last-known cluster state; node term 30, last-accepted version 279918 in term 30; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.15/discovery-troubleshooting.html
[2024-08-14T13:25:03,225][WARN ][o.e.c.c.ClusterFormationFailureHelper] [cvelk6] master not discovered or elected yet, an election requires a node with id [MsUT9oCuSauu5MSea2E_Wg], have only discovered non-quorum [{cvelk6}{jkx38DGkQcqlcpxZy6Ih2w}{ASnoIqWTTVunVXkR7IOBWw}{cvelk6}{204.141.9.13}{204.141.9.13:9300}{dimrs}{8.15.0}{7000099-8512000}, {cvelk5}{apRS6ewWRWKWA8_6HsteiQ}{VktzXDvBRcOzcZrwvTuz9Q}{cvelk5}{204.141.9.12}{204.141.9.12:9300}{dimrs}{8.15.0}{7000099-8512000}]; discovery will continue using [204.141.9.12:9300, 127.0.1.1:9300] from hosts providers and [{cvelk6}{jkx38DGkQcqlcpxZy6Ih2w}{ASnoIqWTTVunVXkR7IOBWw}{cvelk6}{204.141.9.13}{204.141.9.13:9300}{dimrs}{8.15.0}{7000099-8512000}] from last-known cluster state; node term 30, last-accepted version 279918 in term 30; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.15/discovery-troubleshooting.html
[2024-08-14T13:25:13,225][WARN ][o.e.n.Node               ] [cvelk6] timed out while waiting for initial discovery state - timeout: 30s
[2024-08-14T13:25:13,227][WARN ][o.e.c.c.ClusterFormationFailureHelper] [cvelk6] master not discovered or elected yet, an election requires a node with id [MsUT9oCuSauu5MSea2E_Wg], have only discovered non-quorum [{cvelk6}{jkx38DGkQcqlcpxZy6Ih2w}{ASnoIqWTTVunVXkR7IOBWw}{cvelk6}{204.141.9.13}{204.141.9.13:9300}{dimrs}{8.15.0}{7000099-8512000}, {cvelk5}{apRS6ewWRWKWA8_6HsteiQ}{VktzXDvBRcOzcZrwvTuz9Q}{cvelk5}{204.141.9.12}{204.141.9.12:9300}{dimrs}{8.15.0}{7000099-8512000}]; discovery will continue using [204.141.9.12:9300, 127.0.1.1:9300] from hosts providers and [{cvelk6}{jkx38DGkQcqlcpxZy6Ih2w}{ASnoIqWTTVunVXkR7IOBWw}{cvelk6}{204.141.9.13}{204.141.9.13:9300}{dimrs}{8.15.0}{7000099-8512000}] from last-known cluster state; node term 30, last-accepted version 279918 in term 30; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.15/discovery-troubleshooting.html
[2024-08-14T13:25:13,235][INFO ][o.e.h.AbstractHttpServerTransport] [cvelk6] publish_address {204.141.9.13:9200}, bound_addresses {[::]:9200}
[2024-08-14T13:25:13,268][INFO ][o.e.n.Node               ] [cvelk6] started {cvelk6}{jkx38DGkQcqlcpxZy6Ih2w}{ASnoIqWTTVunVXkR7IOBWw}{cvelk6}{204.141.9.13}{204.141.9.13:9300}{dimrs}{8.15.0}{7000099-8512000}{transform.config_version=10.0.0, xpack.installed=true, ml.config_version=12.0.0}
[2024-08-14T13:25:17,292][INFO ][o.e.x.s.a.RealmsAuthenticator] [cvelk6] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2024-08-14T13:25:17,837][INFO ][o.e.x.s.a.RealmsAuthenticator] [cvelk6] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2024-08-14T13:25:18,340][INFO ][o.e.x.s.a.RealmsAuthenticator] [cvelk6] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2024-08-14T13:25:18,341][INFO ][o.e.x.s.a.RealmsAuthenticator] [cvelk6] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2024-08-14T13:25:22,389][INFO ][o.e.x.s.a.RealmsAuthenticator] [cvelk6] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2024-08-14T13:25:22,922][INFO ][o.e.x.s.a.RealmsAuthenticator] [cvelk6] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2024-08-14T13:25:23,228][WARN ][o.e.c.c.ClusterFormationFailureHelper] [cvelk6] master not discovered or elected yet, an election requires a node with id [MsUT9oCuSauu5MSea2E_Wg], have only discovered non-quorum [{cvelk6}{jkx38DGkQcqlcpxZy6Ih2w}{ASnoIqWTTVunVXkR7IOBWw}{cvelk6}{204.141.9.13}{204.141.9.13:9300}{dimrs}{8.15.0}{7000099-8512000}, {cvelk5}{apRS6ewWRWKWA8_6HsteiQ}{VktzXDvBRcOzcZrwvTuz9Q}{cvelk5}{204.141.9.12}{204.141.9.12:9300}{dimrs}{8.15.0}{7000099-8512000}]; discovery will continue using [204.141.9.12:9300, 127.0.1.1:9300] from hosts providers and [{cvelk6}{jkx38DGkQcqlcpxZy6Ih2w}{ASnoIqWTTVunVXkR7IOBWw}{cvelk6}{204.141.9.13}{204.141.9.13:9300}{dimrs}{8.15.0}{7000099-8512000}] from last-known cluster state; node term 30, last-accepted version 279918 in term 30; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.15/discovery-troubleshooting.html

rachelyang · August 14, 2024, 3:37pm

These are the logs after I restarted the elasticsearch service.
If you need more information, please let me know.

rachelyang · August 14, 2024, 6:20pm

I manually create a new index, then ES instance comes back.

Topic		Replies	Views
Shard failed recovery upon rolling upgrade due to storeTermVector error Elasticsearch	1	371	April 6, 2022
Rolling Upgrade from 6.8.0 to 7.3.1 Failed to Join the Cluster Elasticsearch	5	652	October 7, 2019
Getting the error "Cluster state has not been recovered yet, cannot write to the [null] index" in 7.3 version Elasticsearch	2	2598	October 23, 2019
Upgrade from 6.8.3 to 7.X.X results in "failed to apply updated cluster state" Elasticsearch	13	1554	December 29, 2019
After upgrading from 7.x to 8.x, Elasticsearch cannot start Elasticsearch	5	66	September 4, 2024

Elasticsearch rolling upgrade failed

Related topics