Data nodes unable to see the data in path.data

Elastic Stack Elasticsearch
1

Hello all,

Initially my Elastic cluster was down after a trial license expired.
I was able to successfully delete the license and revert back to the basic one.

After that, all the nodes are up with a master. Nonetheless, the data nodes are unable to see the data in path.data. Please note that the path.data is correctly set in all data nodes inside the YML files for all the data nodes.

I also tried to do cluster re-route and shard re-assignement with no luck.

Can you please provide me with some guidance on how I can make the nodes correctly see the indices? Thank you in advance.

Some insights on the status of the cluster:

-XGET "https://localhost:9200/_cat/nodes"

10.0.1.20 11 67 0 0.01 0.20 0.31 cdfhilrstw - 10.0.1.20
10.0.1.18  7 98 0 0.01 0.17 0.23 ilmr       * 10.0.1.18
10.0.1.25 59 67 0 0.00 0.13 0.17 cdfhilrstw - 10.0.1.25
10.0.1.23 48 67 0 0.00 0.10 0.14 cdfhilrstw - 10.0.1.23
10.0.1.22  3 61 0 0.00 0.11 0.16 cdfhilrstw - 10.0.1.22
10.0.1.19 15 99 0 0.00 0.09 0.10 ilmr       - 10.0.1.19
10.0.1.21 30 61 0 0.01 0.15 0.17 cdfhilrstw - 10.0.1.21
10.0.1.24  6 61 0 0.00 0.12 0.15 cdfhilrstw - 10.0.1.24
10.0.1.17 40 99 0 0.02 0.10 0.10 ilmr       - 10.0.1.17

-XGET "https://localhost:9200/_cluster/health?pretty

{
  "cluster_name" : "elk-cluster",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 9,
  "number_of_data_nodes" : 6,
  "active_primary_shards" : 9,
  "active_shards" : 18,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 2556,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 0.6993006993006993
}
2

Welcome to our community! :smiley:

What does your elasticsearch.yml look like? What do the Elasticsearch logs show?

3

Thank you for your prompt response. Below I am providing more information.

elasticsearch.yml

bootstrap.memory_lock: false
cluster.initial_master_nodes: 10.0.1.17,10.0.1.18,10.0.1.19
cluster.name: elk-cluster
discovery.seed_hosts: 10.0.1.17:9300,10.0.1.18:9300,10.0.1.19:9300
http.host: 0.0.0.0
http.port: 9200
network.host: 10.0.1.17
node.data: false
node.master: true
transport.host: 0.0.0.0
transport.port: 9300



node.name: 10.0.1.17

#################################### Paths ####################################

# Path to directory containing configuration (this file and logging.yml):

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

path.repo: ['/mnt/OSShare']

action.auto_create_index: true

xpack.security.enabled: true

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: "certificate"
xpack.security.transport.ssl.keystore.path: "/etc/elasticsearch/certs/my-keystore.p12"
xpack.security.transport.ssl.truststore.path: "/etc/elasticsearch/certs/my-ca.p12"

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: "/etc/elasticsearch/certs/my-keystore.p12"
xpack.security.http.ssl.truststore.path: "/etc/elasticsearch/certs/my-ca.p12"
xpack.license.self_generated.type: basic

Few more information that explain more about my situation ("reason" : "CLUSTER_RECOVERED" and "can_allocate" : "no_valid_shard_copy"):

-XGET "https://localhost:9200/_cluster/allocation/explain?pretty"

{
  "note" : "No shard was specified in the explain API request, so this response explains a randomly chosen unassigned shard. There may be other unassigned shards in this cluster which cannot be assigned for different reasons. It may not be possible to assign this shard until one of the other shards is assigned correctly. To explain the allocation of other shards (whether assigned or unassigned) you must specify the target shard in the request to this API.",
  "index" : "2020-filebeat-system-7.6.2-2021.06.24",
  "shard" : 0,
  "primary" : true,
  "current_state" : "unassigned",
  "unassigned_info" : {
    "reason" : "CLUSTER_RECOVERED",
    "at" : "2022-11-21T02:53:18.252Z",
    "last_allocation_status" : "no_valid_shard_copy"
  },
  "can_allocate" : "no_valid_shard_copy",
  "allocate_explanation" : "cannot allocate because a previous copy of the primary shard existed but can no longer be found on the nodes in the cluster",
  "node_allocation_decisions" : [
    {
      "node_id" : "8ZzV1BsPSL6a9tYaK5yXTQ",
      "node_name" : "10.0.1.20",
      "transport_address" : "10.0.1.20:9300",
      "node_attributes" : {
        "ml.machine_memory" : "63206785024",
        "ml.max_open_jobs" : "512",
        "xpack.installed" : "true",
        "ml.max_jvm_size" : "34359738368",
        "transform.node" : "true"
      },
      "node_decision" : "no",
      "store" : {
        "found" : false
      }
    },
...
4

Can you post your logs from Elasticsearch please?

5

/var/log/elasticsearch/elk-cluster.log

[2022-11-21T02:42:38,446][INFO ][o.e.t.ClusterConnectionManager] [10.0.1.17] transport connection to [{10.0.1.25}{RIWCIcnHQ3-r64yF9ICdoQ}{K9dFSPJuTxqUJ7NPG0QNvw}{10.0.1.25}{10.0.1.25:9300}{cdfhilrstw}] closed by remote
[2022-11-21T02:42:38,514][INFO ][o.e.c.s.ClusterApplierService] [10.0.1.17] removed {{10.0.1.25}{RIWCIcnHQ3-r64yF9ICdoQ}{K9dFSPJuTxqUJ7NPG0QNvw}{10.0.1.25}{10.0.1.25:9300}{cdfhilrstw}}, term: 16, version: 7568374, reason: ApplyCommitRequest{term=16, version=7568374, sourceNode={10.0.1.18}{Ed4va8FURTS6pY0skuarYQ}{8PAbR-eqTGGNdYUonAlxxQ}{10.0.1.18}{10.0.1.18:9300}{ilmr}{ml.machine_memory=31550173184, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=27917287424, transform.node=false}}
[2022-11-21T02:43:15,693][INFO ][o.e.c.s.ClusterApplierService] [10.0.1.17] added {{10.0.1.21}{UirUyJmcRYGlMPg1237cfg}{gjRwEymqTtaPI-3zXAkI4w}{10.0.1.21}{10.0.1.21:9300}{cdfhilrstw}}, term: 16, version: 7568376, reason: ApplyCommitRequest{term=16, version=7568376, sourceNode={10.0.1.18}{Ed4va8FURTS6pY0skuarYQ}{8PAbR-eqTGGNdYUonAlxxQ}{10.0.1.18}{10.0.1.18:9300}{ilmr}{ml.machine_memory=31550173184, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=27917287424, transform.node=false}}
[2022-11-21T02:43:23,064][INFO ][o.e.c.s.ClusterApplierService] [10.0.1.17] added {{10.0.1.22}{DHN8J1n8RC6KR67bptS_2A}{ayT1s0ApTFy7EJeDb9AHbQ}{10.0.1.22}{10.0.1.22:9300}{cdfhilrstw}}, term: 16, version: 7568383, reason: ApplyCommitRequest{term=16, version=7568383, sourceNode={10.0.1.18}{Ed4va8FURTS6pY0skuarYQ}{8PAbR-eqTGGNdYUonAlxxQ}{10.0.1.18}{10.0.1.18:9300}{ilmr}{ml.machine_memory=31550173184, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=27917287424, transform.node=false}}
[2022-11-21T02:43:27,455][INFO ][o.e.c.s.ClusterApplierService] [10.0.1.17] added {{10.0.1.20}{8ZzV1BsPSL6a9tYaK5yXTQ}{JvyZwuJISICdWkjXxDQ6hQ}{10.0.1.20}{10.0.1.20:9300}{cdfhilrstw}}, term: 16, version: 7568396, reason: ApplyCommitRequest{term=16, version=7568396, sourceNode={10.0.1.18}{Ed4va8FURTS6pY0skuarYQ}{8PAbR-eqTGGNdYUonAlxxQ}{10.0.1.18}{10.0.1.18:9300}{ilmr}{ml.machine_memory=31550173184, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=27917287424, transform.node=false}}
[2022-11-21T02:43:30,904][INFO ][o.e.c.s.ClusterApplierService] [10.0.1.17] added {{10.0.1.23}{XkfWuHCzQMiCYWKPCdyGSw}{tEbCwLtIT_yy-aSua78QiQ}{10.0.1.23}{10.0.1.23:9300}{cdfhilrstw}, {10.0.1.24}{OiMqpYzfRmmPcOYLvFf8GA}{xPbEeKM0QQuo4r9U4-M92A}{10.0.1.24}{10.0.1.24:9300}{cdfhilrstw}}, term: 16, version: 7568398, reason: ApplyCommitRequest{term=16, version=7568398, sourceNode={10.0.1.18}{Ed4va8FURTS6pY0skuarYQ}{8PAbR-eqTGGNdYUonAlxxQ}{10.0.1.18}{10.0.1.18:9300}{ilmr}{ml.machine_memory=31550173184, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=27917287424, transform.node=false}}
[2022-11-21T02:43:37,658][INFO ][o.e.c.s.ClusterApplierService] [10.0.1.17] added {{10.0.1.25}{RIWCIcnHQ3-r64yF9ICdoQ}{OAY50NBFRum77WTdwKPyAw}{10.0.1.25}{10.0.1.25:9300}{cdfhilrstw}}, term: 16, version: 7568408, reason: ApplyCommitRequest{term=16, version=7568408, sourceNode={10.0.1.18}{Ed4va8FURTS6pY0skuarYQ}{8PAbR-eqTGGNdYUonAlxxQ}{10.0.1.18}{10.0.1.18:9300}{ilmr}{ml.machine_memory=31550173184, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=27917287424, transform.node=false}}
[2022-11-21T02:43:55,342][INFO ][o.e.c.s.ClusterSettings  ] [10.0.1.17] updating [cluster.routing.allocation.enable] from [all] to [primaries]
[2022-11-21T02:43:55,342][INFO ][o.e.c.s.ClusterSettings  ] [10.0.1.17] updating [cluster.routing.allocation.enable] from [all] to [primaries]
[2022-11-21T02:44:46,117][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [10.0.1.17] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.0.1.17:9200, remoteAddress=/10.0.1.20:47048}
[2022-11-21T02:52:20,034][INFO ][o.e.t.ClusterConnectionManager] [10.0.1.17] transport connection to [{10.0.1.19}{vUFC79mARjSqkQblTgb7vg}{oVFTEpAVS2iUZ8MihG1JFg}{10.0.1.19}{10.0.1.19:9300}{ilmr}] closed by remote
[2022-11-21T02:52:20,089][INFO ][o.e.c.s.ClusterApplierService] [10.0.1.17] removed {{10.0.1.19}{vUFC79mARjSqkQblTgb7vg}{oVFTEpAVS2iUZ8MihG1JFg}{10.0.1.19}{10.0.1.19:9300}{ilmr}}, term: 16, version: 7569676, reason: ApplyCommitRequest{term=16, version=7569676, sourceNode={10.0.1.18}{Ed4va8FURTS6pY0skuarYQ}{8PAbR-eqTGGNdYUonAlxxQ}{10.0.1.18}{10.0.1.18:9300}{ilmr}{ml.machine_memory=31550173184, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=27917287424, transform.node=false}}
[2022-11-21T02:52:22,582][INFO ][o.e.t.ClusterConnectionManager] [10.0.1.17] transport connection to [{10.0.1.18}{Ed4va8FURTS6pY0skuarYQ}{8PAbR-eqTGGNdYUonAlxxQ}{10.0.1.18}{10.0.1.18:9300}{ilmr}] closed by remote
[2022-11-21T02:52:22,584][INFO ][o.e.c.c.Coordinator      ] [10.0.1.17] master node [{10.0.1.18}{Ed4va8FURTS6pY0skuarYQ}{8PAbR-eqTGGNdYUonAlxxQ}{10.0.1.18}{10.0.1.18:9300}{ilmr}] disconnected, restarting discovery
[2022-11-21T02:52:22,585][INFO ][o.e.c.s.ClusterApplierService] [10.0.1.17] master node changed {previous [{10.0.1.18}{Ed4va8FURTS6pY0skuarYQ}{8PAbR-eqTGGNdYUonAlxxQ}{10.0.1.18}{10.0.1.18:9300}{ilmr}], current []}, term: 16, version: 7569676, reason: becoming candidate: onLeaderFailure
[2022-11-21T02:52:22,588][WARN ][o.e.c.NodeConnectionsService] [10.0.1.17] failed to connect to {10.0.1.18}{Ed4va8FURTS6pY0skuarYQ}{8PAbR-eqTGGNdYUonAlxxQ}{10.0.1.18}{10.0.1.18:9300}{ilmr}{ml.machine_memory=31550173184, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=27917287424, transform.node=false} (tried [1] times)
org.elasticsearch.transport.ConnectTransportException: [10.0.1.18][10.0.1.18:9300] connect_exception
	at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1047) ~[elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$0(ActionListener.java:279) ~[elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.core.CompletableContext.lambda$addListener$0(CompletableContext.java:31) ~[elasticsearch-core-7.17.0.jar:7.17.0]
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?]
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) ~[?:?]
	at org.elasticsearch.core.CompletableContext.completeExceptionally(CompletableContext.java:46) ~[elasticsearch-core-7.17.0.jar:7.17.0]
	at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$addListener$0(Netty4TcpChannel.java:58) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:571) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:550) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:609) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117) ~[?:?]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321) ~[?:?]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:707) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: 10.0.1.18/10.0.1.18:9300
Caused by: java.net.ConnectException: Connection refused
	at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
	at sun.nio.ch.Net.pollConnectNow(Net.java:672) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946) ~[?:?]
	at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330) ~[?:?]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) ~[?:?]
	... 7 more
[2022-11-21T02:52:27,435][INFO ][o.e.n.Node               ] [10.0.1.17] stopping ...
[2022-11-21T02:52:27,438][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [10.0.1.17] [controller/5930] [Main.cc@174] ML controller exiting
[2022-11-21T02:52:27,438][INFO ][o.e.x.m.p.NativeController] [10.0.1.17] Native controller process has stopped - no new native processes can be started
[2022-11-21T02:52:27,439][INFO ][o.e.x.w.WatcherService   ] [10.0.1.17] stopping watch service, reason [shutdown initiated]
[2022-11-21T02:52:27,439][INFO ][o.e.x.w.WatcherLifeCycleService] [10.0.1.17] watcher has stopped and shutdown
[2022-11-21T02:52:27,451][INFO ][o.e.n.Node               ] [10.0.1.17] stopped
[2022-11-21T02:52:27,452][INFO ][o.e.n.Node               ] [10.0.1.17] closing ...
[2022-11-21T02:52:27,464][INFO ][o.e.n.Node               ] [10.0.1.17] closed
[2022-11-21T02:52:33,127][INFO ][o.e.n.Node               ] [10.0.1.17] version[7.17.0], pid[8219], build[default/deb/bee86328705acaa9a6daede7140defd4d9ec56bd/2022-01-28T08:36:04.875279988Z], OS[Linux/5.15.0-48-generic/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.1/17.0.1+12]
[2022-11-21T02:52:33,131][INFO ][o.e.n.Node               ] [10.0.1.17] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2022-11-21T02:52:33,131][INFO ][o.e.n.Node               ] [10.0.1.17] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -Xms26g, -Xmx26g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=13958643712, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb, -Des.bundled_jdk=true]
[2022-11-21T02:52:34,726][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [aggs-matrix-stats]
[2022-11-21T02:52:34,726][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [analysis-common]
[2022-11-21T02:52:34,726][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [constant-keyword]
[2022-11-21T02:52:34,726][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [frozen-indices]
[2022-11-21T02:52:34,727][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [ingest-common]
[2022-11-21T02:52:34,727][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [ingest-geoip]
[2022-11-21T02:52:34,727][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [ingest-user-agent]
[2022-11-21T02:52:34,727][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [kibana]
[2022-11-21T02:52:34,727][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [lang-expression]
[2022-11-21T02:52:34,728][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [lang-mustache]
[2022-11-21T02:52:34,728][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [lang-painless]
[2022-11-21T02:52:34,728][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [legacy-geo]
[2022-11-21T02:52:34,728][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [mapper-extras]
[2022-11-21T02:52:34,728][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [mapper-version]
[2022-11-21T02:52:34,729][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [parent-join]
[2022-11-21T02:52:34,729][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [percolator]
[2022-11-21T02:52:34,729][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [rank-eval]
[2022-11-21T02:52:34,729][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [reindex]
[2022-11-21T02:52:34,729][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [repositories-metering-api]
[2022-11-21T02:52:34,729][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [repository-encrypted]
[2022-11-21T02:52:34,730][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [repository-url]
[2022-11-21T02:52:34,730][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [runtime-fields-common]
[2022-11-21T02:52:34,730][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [search-business-rules]
[2022-11-21T02:52:34,730][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [searchable-snapshots]
[2022-11-21T02:52:34,730][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [snapshot-repo-test-kit]
[2022-11-21T02:52:34,730][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [spatial]
[2022-11-21T02:52:34,731][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [systemd]
[2022-11-21T02:52:34,731][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [transform]
[2022-11-21T02:52:34,731][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [transport-netty4]
[2022-11-21T02:52:34,731][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [unsigned-long]
[2022-11-21T02:52:34,731][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [vector-tile]
[2022-11-21T02:52:34,732][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [vectors]
[2022-11-21T02:52:34,732][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [wildcard]
[2022-11-21T02:52:34,732][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [x-pack-aggregate-metric]
[2022-11-21T02:52:34,732][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [x-pack-analytics]
[2022-11-21T02:52:34,732][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [x-pack-async]
[2022-11-21T02:52:34,732][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [x-pack-async-search]
[2022-11-21T02:52:34,733][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [x-pack-autoscaling]
[2022-11-21T02:52:34,733][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [x-pack-ccr]
[2022-11-21T02:52:34,733][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [x-pack-core]
[2022-11-21T02:52:34,733][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [x-pack-data-streams]
[2022-11-21T02:52:34,733][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [x-pack-deprecation]
[2022-11-21T02:52:34,733][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [x-pack-enrich]
[2022-11-21T02:52:34,734][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [x-pack-eql]
[2022-11-21T02:52:34,734][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [x-pack-fleet]
[2022-11-21T02:52:34,734][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [x-pack-graph]
[2022-11-21T02:52:34,734][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [x-pack-identity-provider]
[2022-11-21T02:52:34,734][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [x-pack-ilm]
[2022-11-21T02:52:34,734][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [x-pack-logstash]
[2022-11-21T02:52:34,735][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [x-pack-ml]
[2022-11-21T02:52:34,735][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [x-pack-monitoring]
[2022-11-21T02:52:34,735][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [x-pack-ql]
[2022-11-21T02:52:34,735][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [x-pack-rollup]
[2022-11-21T02:52:34,735][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [x-pack-security]
[2022-11-21T02:52:34,735][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [x-pack-shutdown]
[2022-11-21T02:52:34,736][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [x-pack-sql]
[2022-11-21T02:52:34,736][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [x-pack-stack]
[2022-11-21T02:52:34,736][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [x-pack-text-structure]
[2022-11-21T02:52:34,736][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [x-pack-voting-only-node]
[2022-11-21T02:52:34,736][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded module [x-pack-watcher]
[2022-11-21T02:52:34,737][INFO ][o.e.p.PluginsService     ] [10.0.1.17] loaded plugin [ingest-attachment]
[2022-11-21T02:52:34,771][INFO ][o.e.e.NodeEnvironment    ] [10.0.1.17] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [47.5gb], net total_space [57.9gb], types [ext4]
[2022-11-21T02:52:34,771][INFO ][o.e.e.NodeEnvironment    ] [10.0.1.17] heap size [26gb], compressed ordinary object pointers [true]
[2022-11-21T02:52:34,898][INFO ][o.e.n.Node               ] [10.0.1.17] node name [10.0.1.17], node ID [71WY3LZUTDCUTI_W9_q4hQ], cluster name [elk-cluster], roles [master, remote_cluster_client, ml, ingest]
[2022-11-21T02:52:38,301][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [10.0.1.17] [controller/8400] [Main.cc@122] controller (64 bit): Version 7.17.0 (Build 447aed7cba032c) Copyright (c) 2022 Elasticsearch BV
[2022-11-21T02:52:38,596][INFO ][o.e.x.s.a.Realms         ] [10.0.1.17] license mode is [trial], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]
[2022-11-21T02:52:38,829][INFO ][o.e.x.s.a.s.FileRolesStore] [10.0.1.17] parsed [3] roles from file [/etc/elasticsearch/roles.yml]
[2022-11-21T02:52:39,033][INFO ][o.e.i.g.ConfigDatabases  ] [10.0.1.17] initialized default databases [[GeoLite2-Country.mmdb, GeoLite2-City.mmdb, GeoLite2-ASN.mmdb]], config databases [[]] and watching [/etc/elasticsearch/ingest-geoip] for changes
[2022-11-21T02:52:39,034][INFO ][o.e.i.g.DatabaseNodeService] [10.0.1.17] initialized database registry, using geoip-databases directory [/tmp/geoip-databases/71WY3LZUTDCUTI_W9_q4hQ]
[2022-11-21T02:52:39,487][INFO ][o.e.t.NettyAllocator     ] [10.0.1.17] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=16mb}]
[2022-11-21T02:52:39,542][INFO ][o.e.d.DiscoveryModule    ] [10.0.1.17] using discovery type [zen] and seed hosts providers [settings]
[2022-11-21T02:52:39,882][INFO ][o.e.g.DanglingIndicesState] [10.0.1.17] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2022-11-21T02:52:40,268][INFO ][o.e.n.Node               ] [10.0.1.17] initialized
[2022-11-21T02:52:40,268][INFO ][o.e.n.Node               ] [10.0.1.17] starting ...
[2022-11-21T02:52:40,271][INFO ][o.e.x.d.l.DeprecationIndexingComponent] [10.0.1.17] deprecation component started
[2022-11-21T02:52:40,349][INFO ][o.e.t.TransportService   ] [10.0.1.17] publish_address {10.0.1.17:9300}, bound_addresses {[::]:9300}
[2022-11-21T02:53:09,407][INFO ][o.e.b.BootstrapChecks    ] [10.0.1.17] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2022-11-21T02:53:09,409][INFO ][o.e.c.c.Coordinator      ] [10.0.1.17] cluster UUID [69IWRtPySDa-5GCEhj2PKg]
[2022-11-21T02:53:09,722][INFO ][o.e.c.s.MasterService    ] [10.0.1.17] elected-as-master ([2] nodes joined)[{10.0.1.19}{vUFC79mARjSqkQblTgb7vg}{aPesA4_KRNG65AYY6jXwGA}{10.0.1.19}{10.0.1.19:9300}{ilmr} elect leader, {10.0.1.17}{71WY3LZUTDCUTI_W9_q4hQ}{nC6CFGMhS3O7gChYrHRQug}{10.0.1.17}{10.0.1.17:9300}{ilmr} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 18, version: 7569677, delta: master node changed {previous [], current [{10.0.1.17}{71WY3LZUTDCUTI_W9_q4hQ}{nC6CFGMhS3O7gChYrHRQug}{10.0.1.17}{10.0.1.17:9300}{ilmr}]}, added {{10.0.1.19}{vUFC79mARjSqkQblTgb7vg}{aPesA4_KRNG65AYY6jXwGA}{10.0.1.19}{10.0.1.19:9300}{ilmr}}
[2022-11-21T02:53:11,705][WARN ][o.e.c.s.MasterService    ] [10.0.1.17] failing [elected-as-master ([2] nodes joined)[{10.0.1.19}{vUFC79mARjSqkQblTgb7vg}{aPesA4_KRNG65AYY6jXwGA}{10.0.1.19}{10.0.1.19:9300}{ilmr} elect leader, {10.0.1.17}{71WY3LZUTDCUTI_W9_q4hQ}{nC6CFGMhS3O7gChYrHRQug}{10.0.1.17}{10.0.1.17:9300}{ilmr} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_]]: failed to commit cluster state version [7569677]
org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: publication failed
	at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication$4.onFailure(Coordinator.java:1772) ~[elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:115) ~[elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:55) ~[elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication.onCompletion(Coordinator.java:1679) ~[elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.cluster.coordination.Publication.onPossibleCompletion(Publication.java:114) ~[elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.cluster.coordination.Publication.cancel(Publication.java:78) ~[elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.cluster.coordination.Coordinator.cancelActivePublication(Coordinator.java:1430) ~[elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.cluster.coordination.Coordinator.becomeCandidate(Coordinator.java:732) ~[elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.cluster.coordination.Coordinator.joinLeaderInTerm(Coordinator.java:561) ~[elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$5(JoinHelper.java:183) ~[elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:341) ~[?:?]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$3.onResponse(SecurityServerTransportInterceptor.java:404) ~[?:?]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$3.onResponse(SecurityServerTransportInterceptor.java:394) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeSystemUser(AuthorizationService.java:620) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:250) ~[?:?]
	at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$1(ServerTransportFilter.java:136) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) ~[elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101) ~[elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:102) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:199) ~[?:?]
	at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:128) ~[?:?]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:415) ~[?:?]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:67) ~[elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:260) ~[elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) ~[elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-7.17.0.jar:7.17.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.elasticsearch.ElasticsearchException: publication cancelled before committing: become candidate: joinLeaderInTerm
	at org.elasticsearch.cluster.coordination.Publication.cancel(Publication.java:75) ~[elasticsearch-7.17.0.jar:7.17.0]
	... 24 more
[2022-11-21T02:53:18,126][INFO ][o.e.c.s.ClusterApplierService] [10.0.1.17] master node changed {previous [], current [{10.0.1.18}{Ed4va8FURTS6pY0skuarYQ}{MYzt5ZFNR1ClQMKZi6EX0g}{10.0.1.18}{10.0.1.18:9300}{ilmr}]}, added {{10.0.1.24}{OiMqpYzfRmmPcOYLvFf8GA}{xPbEeKM0QQuo4r9U4-M92A}{10.0.1.24}{10.0.1.24:9300}{cdfhilrstw}, {10.0.1.18}{Ed4va8FURTS6pY0skuarYQ}{MYzt5ZFNR1ClQMKZi6EX0g}{10.0.1.18}{10.0.1.18:9300}{ilmr}, {10.0.1.20}{8ZzV1BsPSL6a9tYaK5yXTQ}{JvyZwuJISICdWkjXxDQ6hQ}{10.0.1.20}{10.0.1.20:9300}{cdfhilrstw}, {10.0.1.22}{DHN8J1n8RC6KR67bptS_2A}{ayT1s0ApTFy7EJeDb9AHbQ}{10.0.1.22}{10.0.1.22:9300}{cdfhilrstw}, {10.0.1.21}{UirUyJmcRYGlMPg1237cfg}{gjRwEymqTtaPI-3zXAkI4w}{10.0.1.21}{10.0.1.21:9300}{cdfhilrstw}, {10.0.1.23}{XkfWuHCzQMiCYWKPCdyGSw}{tEbCwLtIT_yy-aSua78QiQ}{10.0.1.23}{10.0.1.23:9300}{cdfhilrstw}, {10.0.1.19}{vUFC79mARjSqkQblTgb7vg}{aPesA4_KRNG65AYY6jXwGA}{10.0.1.19}{10.0.1.19:9300}{ilmr}, {10.0.1.25}{RIWCIcnHQ3-r64yF9ICdoQ}{OAY50NBFRum77WTdwKPyAw}{10.0.1.25}{10.0.1.25:9300}{cdfhilrstw}}, term: 19, version: 7569678, reason: ApplyCommitRequest{term=19, version=7569678, sourceNode={10.0.1.18}{Ed4va8FURTS6pY0skuarYQ}{MYzt5ZFNR1ClQMKZi6EX0g}{10.0.1.18}{10.0.1.18:9300}{ilmr}{ml.machine_memory=31550173184, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=27917287424, transform.node=false}}
[2022-11-21T02:53:18,230][INFO ][o.e.h.AbstractHttpServerTransport] [10.0.1.17] publish_address {10.0.1.17:9200}, bound_addresses {[::]:9200}
[2022-11-21T02:53:18,231][INFO ][o.e.n.Node               ] [10.0.1.17] started
[2022-11-21T02:53:21,004][INFO ][o.e.c.s.ClusterSettings  ] [10.0.1.17] updating [cluster.routing.allocation.enable] from [all] to [primaries]
[2022-11-21T02:53:21,004][INFO ][o.e.c.s.ClusterSettings  ] [10.0.1.17] updating [cluster.routing.allocation.enable] from [all] to [primaries]
[2022-11-21T02:53:21,004][INFO ][o.e.c.s.ClusterSettings  ] [10.0.1.17] updating [xpack.monitoring.collection.enabled] from [false] to [true]
[2022-11-21T02:53:21,400][INFO ][o.e.l.LicenseService     ] [10.0.1.17] license [b7aa7039-855e-42b1-b2b0-aefcbc1108af] mode [basic] - valid
[2022-11-21T02:53:21,401][INFO ][o.e.x.s.a.Realms         ] [10.0.1.17] license mode is [basic], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]
[2022-11-21T02:53:21,401][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [10.0.1.17] Active license is now [BASIC]; Security is enabled
[2022-11-21T02:53:21,917][INFO ][o.e.x.s.a.TokenService   ] [10.0.1.17] refresh keys
[2022-11-21T02:53:22,028][INFO ][o.e.x.s.a.TokenService   ] [10.0.1.17] refreshed keys
[2022-11-21T03:14:29,055][INFO ][o.e.c.s.ClusterSettings  ] [10.0.1.17] updating [cluster.routing.allocation.enable] from [primaries] to [all]
[2022-11-21T03:14:29,055][INFO ][o.e.c.s.ClusterSettings  ] [10.0.1.17] updating [cluster.routing.allocation.enable] from [primaries] to [all]
[2022-11-21T03:30:54,404][INFO ][o.e.t.ClusterConnectionManager] [10.0.1.17] transport connection to [{10.0.1.20}{8ZzV1BsPSL6a9tYaK5yXTQ}{JvyZwuJISICdWkjXxDQ6hQ}{10.0.1.20}{10.0.1.20:9300}{cdfhilrstw}] closed by remote
[2022-11-21T03:30:54,496][INFO ][o.e.c.s.ClusterApplierService] [10.0.1.17] removed {{10.0.1.20}{8ZzV1BsPSL6a9tYaK5yXTQ}{JvyZwuJISICdWkjXxDQ6hQ}{10.0.1.20}{10.0.1.20:9300}{cdfhilrstw}}, term: 19, version: 7573475, reason: ApplyCommitRequest{term=19, version=7573475, sourceNode={10.0.1.18}{Ed4va8FURTS6pY0skuarYQ}{MYzt5ZFNR1ClQMKZi6EX0g}{10.0.1.18}{10.0.1.18:9300}{ilmr}{ml.machine_memory=31550173184, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=27917287424, transform.node=false}}
[2022-11-21T03:31:14,744][INFO ][o.e.c.s.ClusterApplierService] [10.0.1.17] added {{10.0.1.20}{DWCdT-JvRb6slN8YdfHPRQ}{I0vjg0l9TM6jFFN4_oIjlw}{10.0.1.20}{10.0.1.20:9300}{cdfhilrstw}}, term: 19, version: 7573476, reason: ApplyCommitRequest{term=19, version=7573476, sourceNode={10.0.1.18}{Ed4va8FURTS6pY0skuarYQ}{MYzt5ZFNR1ClQMKZi6EX0g}{10.0.1.18}{10.0.1.18:9300}{ilmr}{ml.machine_memory=31550173184, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=27917287424, transform.node=false}}
[2022-11-21T03:32:02,889][INFO ][o.e.t.ClusterConnectionManager] [10.0.1.17] transport connection to [{10.0.1.20}{DWCdT-JvRb6slN8YdfHPRQ}{I0vjg0l9TM6jFFN4_oIjlw}{10.0.1.20}{10.0.1.20:9300}{cdfhilrstw}] closed by remote
[2022-11-21T03:32:02,953][INFO ][o.e.c.s.ClusterApplierService] [10.0.1.17] removed {{10.0.1.20}{DWCdT-JvRb6slN8YdfHPRQ}{I0vjg0l9TM6jFFN4_oIjlw}{10.0.1.20}{10.0.1.20:9300}{cdfhilrstw}}, term: 19, version: 7573485, reason: ApplyCommitRequest{term=19, version=7573485, sourceNode={10.0.1.18}{Ed4va8FURTS6pY0skuarYQ}{MYzt5ZFNR1ClQMKZi6EX0g}{10.0.1.18}{10.0.1.18:9300}{ilmr}{ml.machine_memory=31550173184, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=27917287424, transform.node=false}}
[2022-11-21T03:32:56,253][INFO ][o.e.c.s.ClusterApplierService] [10.0.1.17] added {{10.0.1.20}{8ZzV1BsPSL6a9tYaK5yXTQ}{kABrGKEFR8uBRFRonul_mQ}{10.0.1.20}{10.0.1.20:9300}{cdfhilrstw}}, term: 19, version: 7573486, reason: ApplyCommitRequest{term=19, version=7573486, sourceNode={10.0.1.18}{Ed4va8FURTS6pY0skuarYQ}{MYzt5ZFNR1ClQMKZi6EX0g}{10.0.1.18}{10.0.1.18:9300}{ilmr}{ml.machine_memory=31550173184, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=27917287424, transform.node=false}}

Please let me know if additional information if needed (example: all log file).
I highly appreciate your help on this, thank you.

6

That suggests that something is up as it's not reading the setting you have for some reason. Are you sure you updated the config and restarted Elasticsearch to pick up your path.data
?

7

The first log that I sent you belongs to a management node. I believe the path.data is the correct one observed in the logs.

I am attaching below logs pertained to a data node:

[2022-11-21T05:05:00,729][WARN ][r.suppressed             ] [10.0.1.20] path: /.kibana_task_manager_7.17.5_001/_pit, params: {index=.kibana_task_manager_7.17.5_001, keep_alive=10m}
org.elasticsearch.action.search.SearchPhaseExecutionException: 
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:725) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:471) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:211) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1048) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:763) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$6(TransportSearchAction.java:399) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:112) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:77) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:487) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:305) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.action.search.TransportOpenPointInTimeAction.doExecute(TransportOpenPointInTimeAction.java:91) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.action.search.TransportOpenPointInTimeAction.doExecute(TransportOpenPointInTimeAction.java:39) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:179) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:190) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:219) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:577) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:571) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:103) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:575) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:571) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authz.interceptor.DlsFlsLicenseRequestInterceptor.intercept(DlsFlsLicenseRequestInterceptor.java:100) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:575) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:571) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:85) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authz.interceptor.ShardSearchRequestInterceptor.intercept(ShardSearchRequestInterceptor.java:26) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:575) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:571) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:85) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:26) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:575) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:571) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:85) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:27) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:575) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:571) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:87) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:575) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:571) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:128) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:571) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:556) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$11(AuthorizationService.java:450) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:967) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:931) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:352) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:113) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:55) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:41) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:1015) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:343) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:443) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:371) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:256) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:138) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.roles(CompositeRolesStore.java:173) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:279) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:144) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:127) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:258) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:186) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:102) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:182) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:124) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:154) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:82) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:95) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:73) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:407) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.action.search.RestOpenPointInTimeAction.lambda$prepareRequest$0(RestOpenPointInTimeAction.java:45) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:109) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$0(SecurityRestFilter.java:90) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.lambda$authenticateAndAttachToContext$2(SecondaryAuthenticator.java:84) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticate(SecondaryAuthenticator.java:94) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticateAndAttachToContext(SecondaryAuthenticator.java:78) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$2(SecurityRestFilter.java:85) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.action.ActionListener$RunBeforeActionListener.onResponse(ActionListener.java:389) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.writeAuthToContext(AuthenticatorChain.java:376) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.finishAuthentication(AuthenticatorChain.java:352) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.maybeLookupRunAsUser(AuthenticatorChain.java:205) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$doAuthenticate$1(AuthenticatorChain.java:128) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:141) [x-pack-core-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$3(AuthenticatorChain.java:175) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$3(RealmsAuthenticator.java:219) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:141) [x-pack-core-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$0(RealmsAuthenticator.java:164) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$authenticateWithCache$1(CachingUsernamePasswordRealm.java:155) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.handleCachedAuthentication(CachingUsernamePasswordRealm.java:242) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$authenticateWithCache$2(CachingUsernamePasswordRealm.java:139) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:113) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:55) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:134) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:105) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$2(RealmsAuthenticator.java:148) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117) [x-pack-core-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.consumeToken(RealmsAuthenticator.java:233) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.authenticate(RealmsAuthenticator.java:84) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:171) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) [x-pack-core-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:165) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) [x-pack-core-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:165) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) [x-pack-core-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:165) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117) [x-pack-core-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.doAuthenticate(AuthenticatorChain.java:143) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:104) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:149) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:127) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:79) [x-pack-security-7.17.0.jar:7.17.0]
	at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:327) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:393) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:245) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:382) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:461) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:357) [elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:32) [transport-netty4-client-7.17.0.jar:7.17.0]
	at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:18) [transport-netty4-client-7.17.0.jar:7.17.0]
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:48) [transport-netty4-client-7.17.0.jar:7.17.0]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1374) [netty-handler-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1237) [netty-handler-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1286) [netty-handler-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.66.Final.jar:4.1.66.Final]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.kibana_task_manager_7.17.5_001][0]]. Consider using `allow_partial_search_results` setting to bypass this error.
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.run(AbstractSearchAsyncAction.java:239) ~[elasticsearch-7.17.0.jar:7.17.0]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:466) [elasticsearch-7.17.0.jar:7.17.0]
	... 192 more

Data node YML file

bootstrap.memory_lock: false
cluster.initial_master_nodes: 10.0.1.17,10.0.1.18,10.0.1.19
cluster.name: elk-cluster
discovery.seed_hosts: 10.0.1.17:9300,10.0.1.18:9300,10.0.1.19:9300
http.host: 0.0.0.0
http.port: 9200
network.host: 10.0.1.20
node.data: true
node.master: false
transport.host: 0.0.0.0
transport.port: 9300



node.name: 10.0.1.20

#################################### Paths ####################################

# Path to directory containing configuration (this file and logging.yml):

path.data: /var/lib/elasticsearch,/mnt/sdb
#path.data: /mnt/tmp

path.logs: /var/log/elasticsearch

path.repo: ['/mnt/OSShare']

action.auto_create_index: true

xpack.security.enabled: true

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: "certificate"
xpack.security.transport.ssl.keystore.path: "/etc/elasticsearch/certs/my-keystore.p12"
xpack.security.transport.ssl.truststore.path: "/etc/elasticsearch/certs/my-ca.p12"

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: "/etc/elasticsearch/certs/http.p12"
#xpack.security.http.ssl.truststore.path: "/etc/elasticsearch/certs/my-ca.p12"
8

What exactly did you do to delete the license? Specifically, what commands did you run?

9

I used this:

curl -k -s -u es_admin:password -XDELETE 'http://localhost:9200/_license'

10

Update: I am providing below the response of the explain API

-XGET 'https://localhost:9200/_cluster/allocation/explain?pretty'

{
  "note" : "No shard was specified in the explain API request, so this response explains a randomly chosen unassigned shard. There may be other unassigned shards in this cluster which cannot be assigned for different reasons. It may not be possible to assign this shard until one of the other shards is assigned correctly. To explain the allocation of other shards (whether assigned or unassigned) you must specify the target shard in the request to this API.",
  "index" : "2020-filebeat-system-7.6.2-2021.02.19",
  "shard" : 0,
  "primary" : true,
  "current_state" : "unassigned",
  "unassigned_info" : {
    "reason" : "CLUSTER_RECOVERED",
    "at" : "2022-11-21T02:53:18.247Z",
    "last_allocation_status" : "no_valid_shard_copy"
  },
  "can_allocate" : "no_valid_shard_copy",
  "allocate_explanation" : "cannot allocate because a previous copy of the primary shard existed but can no longer be found on the nodes in the cluster",
  "node_allocation_decisions" : [
    {
      "node_id" : "DHN8J1n8RC6KR67bptS_2A",
      "node_name" : "10.0.1.22",
      "transport_address" : "10.0.1.22:9300",
      "node_attributes" : {
        "ml.machine_memory" : "63206793216",
        "ml.max_open_jobs" : "512",
        "xpack.installed" : "true",
        "ml.max_jvm_size" : "34359738368",
        "transform.node" : "true"
      },
      "node_decision" : "no",
      "store" : {
        "found" : false
      }
    },
    {
      "node_id" : "OiMqpYzfRmmPcOYLvFf8GA",
      "node_name" : "10.0.1.24",
      "transport_address" : "10.0.1.24:9300",
      "node_attributes" : {
        "ml.machine_memory" : "63206793216",
        "ml.max_open_jobs" : "512",
        "xpack.installed" : "true",
        "ml.max_jvm_size" : "34359738368",
        "transform.node" : "true"
      },
      "node_decision" : "no",
      "store" : {
        "found" : false
      }
    },
    {
      "node_id" : "RIWCIcnHQ3-r64yF9ICdoQ",
      "node_name" : "10.0.1.25",
      "transport_address" : "10.0.1.25:9300",
      "node_attributes" : {
        "ml.machine_memory" : "63206785024",
        "ml.max_open_jobs" : "512",
        "xpack.installed" : "true",
        "ml.max_jvm_size" : "34359738368",
        "transform.node" : "true"
      },
      "node_decision" : "no",
      "store" : {
        "found" : false
      }
    },
    {
      "node_id" : "UirUyJmcRYGlMPg1237cfg",
      "node_name" : "10.0.1.21",
      "transport_address" : "10.0.1.21:9300",
      "node_attributes" : {
        "ml.machine_memory" : "63206789120",
        "ml.max_open_jobs" : "512",
        "xpack.installed" : "true",
        "ml.max_jvm_size" : "34359738368",
        "transform.node" : "true"
      },
      "node_decision" : "no",
      "store" : {
        "found" : false
      }
    },
    {
      "node_id" : "XkfWuHCzQMiCYWKPCdyGSw",
      "node_name" : "10.0.1.23",
      "transport_address" : "10.0.1.23:9300",
      "node_attributes" : {
        "ml.machine_memory" : "63206793216",
        "ml.max_open_jobs" : "512",
        "xpack.installed" : "true",
        "ml.max_jvm_size" : "34359738368",
        "transform.node" : "true"
      },
      "node_decision" : "no",
      "store" : {
        "found" : false
      }
    }
  ]
}
11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.