SocketTimeout Exception issue reported for Elasticsearch Node with moderate resource usage

The Elasticsearch Cluster with the reported issue includes 86 data nodes and 3 dedicated master nodes, each with 32 GB heap 64 to 128GB OS memory, and between 16 ~ 32 CPU cores per node.

We have two ES clusters with similar setup and same data, we only observe issue in one of the cluster.

Here is the detail of a sample error log on the client side when the issue occur, es-hadoop client library is used.

[2024-06-05 20:48:20,417] ERROR Node [10.246.13.230:9200] failed (java.net.SocketTimeoutException: Read timed out); selected next node [10.246.12.178:9200] (org.elasticsearch.hadoop.rest.NetworkClient)
[2024-06-05 21:07:40,373] ERROR Exception in task 9.0 in stage 551.0 (TID 49607) (org.apache.spark.executor.Executor)
org.elasticsearch.hadoop.EsHadoopException: Could not get a Transport from the Transport Pool for host [10.246.13.229:9200]
        at org.elasticsearch.hadoop.rest.pooling.PooledHttpTransportFactory.borrowFrom(PooledHttpTransportFactory.java:110) ~[spark-job-1.1.35.jar:?]
        at org.elasticsearch.hadoop.rest.pooling.PooledHttpTransportFactory.create(PooledHttpTransportFactory.java:56) ~[spark-job-1.1.35.jar:?]
        at org.elasticsearch.hadoop.rest.NetworkClient.selectNextNode(NetworkClient.java:99) ~[spark-job-1.1.35.jar:?]
        at org.elasticsearch.hadoop.rest.NetworkClient.<init>(NetworkClient.java:82) ~[spark-job-1.1.35.jar:?]
        at org.elasticsearch.hadoop.rest.NetworkClient.<init>(NetworkClient.java:58) ~[spark-job-1.1.35.jar:?]
        at org.elasticsearch.hadoop.rest.RestClient.<init>(RestClient.java:110) ~[spark-job-1.1.35.jar:?]
        at org.elasticsearch.hadoop.rest.InitializationUtils.filterNonDataNodesIfNeeded(InitializationUtils.java:154) ~[spark-job-1.1.35.jar:?]
        at org.elasticsearch.hadoop.rest.RestService.createWriter(RestService.java:587) ~[spark-job-1.1.35.jar:?]
        at org.elasticsearch.spark.rdd.EsRDDWriter.write(EsRDDWriter.scala:71) ~[spark-job-1.1.35.jar:?]
        at org.elasticsearch.spark.rdd.EsSpark$.$anonfun$doSaveToEs$1(EsSpark.scala:108) ~[spark-job-1.1.35.jar:?]
        at org.elasticsearch.spark.rdd.EsSpark$.$anonfun$doSaveToEs$1$adapted(EsSpark.scala:108) ~[spark-job-1.1.35.jar:?]
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) ~[spark-core_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.scheduler.Task.run(Task.scala:136) ~[spark-core_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548) ~[spark-core_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504) ~[spark-core_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551) ~[spark-core_2.12-3.3.0.jar:3.3.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
        at java.lang.Thread.run(Thread.java:829) ~[?:?]
        at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: org.elasticsearch.hadoop.EsHadoopException: java.net.SocketTimeoutException: Read timed out
        at org.elasticsearch.hadoop.mr.security.HadoopUser.doAs(HadoopUser.java:68) ~[spark-job-1.1.35.jar:?]
        at org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport.execute(CommonsHttpTransport.java:681) ~[spark-job-1.1.35.jar:?]
        at org.elasticsearch.hadoop.rest.pooling.TransportPool.validate(TransportPool.java:102) ~[spark-job-1.1.35.jar:?]
        at org.elasticsearch.hadoop.rest.pooling.TransportPool.borrowTransport(TransportPool.java:131) ~[spark-job-1.1.35.jar:?]
        at org.elasticsearch.hadoop.rest.pooling.PooledHttpTransportFactory.borrowFrom(PooledHttpTransportFactory.java:107) ~[spark-job-1.1.35.jar:?]
        ... 18 more
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method) ~[?:?]
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:115) ~[?:?]
        at java.net.SocketInputStream.read(SocketInputStream.java:168) ~[?:?]
        at java.net.SocketInputStream.read(SocketInputStream.java:140) ~[?:?]
        at sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:484) ~[?:?]
        at sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:478) ~[?:?]
        at sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70) ~[?:?]
        at sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1459) ~[?:?]
        at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1070) ~[?:?]
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:252) ~[?:?]
        at java.io.BufferedInputStream.read(BufferedInputStream.java:271) ~[?:?]
        at org.elasticsearch.hadoop.thirdparty.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:77) ~[spark-job-1.1.35.jar:?]
        at org.elasticsearch.hadoop.thirdparty.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:105) ~[spark-job-1.1.35.jar:?]
        at org.elasticsearch.hadoop.thirdparty.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1115) ~[spark-job-1.1.35.jar:?]
        at org.elasticsearch.hadoop.thirdparty.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1832) ~[spark-job-1.1.35.jar:?]
        at org.elasticsearch.hadoop.thirdparty.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1590) ~[spark-job-1.1.35.jar:?]
        at org.elasticsearch.hadoop.thirdparty.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:995) ~[spark-job-1.1.35.jar:?]
        at org.elasticsearch.hadoop.thirdparty.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:397) ~[spark-job-1.1.35.jar:?]
        at org.elasticsearch.hadoop.thirdparty.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170) ~[spark-job-1.1.35.jar:?]
        at org.elasticsearch.hadoop.thirdparty.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396) ~[spark-job-1.1.35.jar:?]
        at org.elasticsearch.hadoop.thirdparty.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:324) ~[spark-job-1.1.35.jar:?]
        at org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport.doExecute(CommonsHttpTransport.java:717) ~[spark-job-1.1.35.jar:?]
        at org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport.access$200(CommonsHttpTransport.java:98) ~[spark-job-1.1.35.jar:?]
        at org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport$2.run(CommonsHttpTransport.java:684) ~[spark-job-1.1.35.jar:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
        at javax.security.auth.Subject.doAs(Subject.java:423) ~[?:?]
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878) ~[hadoop-client-api-3.3.3.jar:?]
        at org.elasticsearch.hadoop.mr.security.HadoopUser.doAs(HadoopUser.java:66) ~[spark-job-1.1.35.jar:?]
        at org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport.execute(CommonsHttpTransport.java:681) ~[spark-job-1.1.35.jar:?]
        at org.elasticsearch.hadoop.rest.pooling.TransportPool.validate(TransportPool.java:102) ~[spark-job-1.1.35.jar:?]
        at org.elasticsearch.hadoop.rest.pooling.TransportPool.borrowTransport(TransportPool.java:131) ~[spark-job-1.1.35.jar:?]
        at org.elasticsearch.hadoop.rest.pooling.PooledHttpTransportFactory.borrowFrom(PooledHttpTransportFactory.java:107) ~[spark-job-1.1.35.jar:?] 

Checking a few of the servers that on the list of servers got timed out, the interesting finding is that the ES nodes are not very busy. Heap pressure is between 50% and 70% and CPU usage is below 30%.

Checking the task list out, there are some of the actions took very long time to complete.

From the task list, we saw high running_time of certain task actions like the following.

  • indices:admin/seq_no/global_checkpoint_sync
  • cluster:monitor/nodes/info
        "U6wDSlMoTF2ipBgvkNPklQ:3186347861" : {
          "node" : "U6wDSlMoTF2ipBgvkNPklQ",
          "id" : 3186347861,
          "type" : "transport",
          "action" : "cluster:monitor/nodes/info",
          "description" : "",
          "start_time" : "2024-06-06T14:06:20.130Z",
          "start_time_in_millis" : 1717682780130,
          "running_time" : "8.3m",
          "running_time_in_nanos" : 500487977174,
          "cancellable" : false,
          "headers" : {
            "X-Opaque-Id" : "[spark] [root] [dummy_spark_job_name]"
          }
        } 

"U6wDSlMoTF2ipBgvkNPklQ:3186411368" : {
          "node" : "U6wDSlMoTF2ipBgvkNPklQ",
          "id" : 3186411368,
          "type" : "direct",
          "action" : "indices:admin/seq_no/global_checkpoint_sync[p]",
          "status" : {
            "phase" : "primary"
          },
          "description" : "GlobalCheckpointSyncAction.Request{shardId=[fdr_6_t2_2024-05-25-000002][10], timeout=1m, index='fdr_6_t2_2024-05-25-000002', waitForActiveShards=1}",
          "start_time" : "2024-06-06T14:10:32.443Z",
          "start_time_in_millis" : 1717683032443,
          "running_time" : "4.1m",
          "running_time_in_nanos" : 248174192644,
          "cancellable" : false,
          "parent_task_id" : "U6wDSlMoTF2ipBgvkNPklQ:3186411367",
          "headers" : {
            "X-Opaque-Id" : "[spark] [root] [dummy_spark_job_name]"
          }
        },

In the server log, noticed the logs for slow transmitting the data to the remote host for the node info request, which seems to be related to the long running_time in the task list.

[2024-06-06T14:21:04,058][WARN ][o.e.t.OutboundHandler    ] [datanode-es-jetstream-prd1-sbq8.us-east1-c.c.spcstg1-sedjetstream.internal]sending transport message [Request{cluster:monitor/nodes/info[n]}{` `2326967899` `}{` `false` `}{` `false` `}{` `false` `}] of size [` `23512` `] on [Netty4TcpChannel{localAddress=/` `10.246` `.` `13.39` `:` `37870` `, remoteAddress=` `10.246` `.` `12.140` `/` `10.246` `.` `12.140` `:` `9300` `, profile=` `default` `}] took [898985ms] which is above the warn threshold of [5000ms] with success [` `false` `]`

`[2024-06-06T14:21:04,058][WARN ][o.e.t.OutboundHandler    ] [datanode-es-jetstream-prd1-sbq8.us-east1-c.c.spcstg1-sedjetstream.internal]sending transport message [Request{cluster:monitor/nodes/info[n]}{` `2326968166` `}{` `false` `}{` `false` `}{` `false` `}] of size [` `23509` `] on [Netty4TcpChannel{localAddress=/` `10.246` `.` `13.39` `:` `37870` `, remoteAddress=` `10.246` `.` `12.140` `/` `10.246` `.` `12.140` `:` `9300` `, profile=` `default` `}] took [893982ms] which is above the warn threshold of [5000ms] with success [` `false` `]`

From the hot thread dump, we could see node info was processed on the transport worker thread which possibly slows down all network operation occurring on the node?

Could this explain the SocketTimeout exception for the bulk indexing request to the same node?

 100.0% [cpu=2.4%, other=97.6%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[datanode-es-jetstream-prd1-gwc6.us-east1-c.c.spcstg1-sedjetstream.internal][transport_worker][T#8]'
     unique snapshot
       java.base@18/java.lang.String.intern(Native Method)
       app//org.elasticsearch.cluster.node.DiscoveryNode.<init>(DiscoveryNode.java:305)
       app//org.elasticsearch.action.support.nodes.BaseNodesRequest$$Lambda$6155/0x0000000801aa7548.read(Unknown Source)
       app//org.elasticsearch.common.io.stream.StreamInput.readArray(StreamInput.java:1023)
       app//org.elasticsearch.common.io.stream.StreamInput.readOptionalArray(StreamInput.java:1029)
       app//org.elasticsearch.action.support.nodes.BaseNodesRequest.<init>(BaseNodesRequest.java:46)
       app//org.elasticsearch.action.admin.cluster.node.info.NodesInfoRequest.<init>(NodesInfoRequest.java:38)
       app//org.elasticsearch.action.admin.cluster.node.info.TransportNodesInfoAction$NodeInfoRequest.<init>(TransportNodesInfoAction.java:102)
       app//org.elasticsearch.action.admin.cluster.node.info.TransportNodesInfoAction$$Lambda$5015/0x00000008017fe080.read(Unknown Source)
       app//org.elasticsearch.transport.RequestHandlerRegistry.newRequest(RequestHandlerRegistry.java:54)
       app//org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:229)
       app//org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:106)
       app//org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:88)
       app//org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:743)
       org.elasticsearch.transport.netty4.Netty4MessageChannelHandler$$Lambda$5916/0x00000008019ff178.accept(Unknown Source)
       app//org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:147)
       app//org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:119)
       app//org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:84)
       org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:71)
       io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
       io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
       io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
       io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
       io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
       io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
       io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
       io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
       io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
       io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
       io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
       io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1374)
       io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1237)
       io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1286)
       io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507)
       io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446)
       io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
       io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
       io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
       io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
       io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
       io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
       io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
       io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
       io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
       io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
       io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620)
       io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583)
       io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
       io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
       io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
       java.base@18/java.lang.Thread.run(Thread.java:833)

What we have also checked

  • We ensure that there is no expensive queries running when the issue occurred.

  • We checked that no snapshotting or data restoring occurred

  • The disk usage of the node affected was less than 60%

  • Each node in the cluster has about 200 open network connections

  • We were suspecting there could be a network problem but not sure what metrics could be checked

We also did some basic network testing between the nodes without SSL, it is about 2 GBytes per second.

MTU is seto 1460 on both server and client.

== Server ==

Accepted connection from 10.246 . 12.69 , port 41704

[ 5 ] local 10.246 . 12.52 port 7575 connected to 10.246 . 12.69 port 41708

[ ID] Interval Transfer Bitrate

[ 5 ] 0.00 - 1.00 sec 1.95 GBytes 16.7 Gbits/sec

[ 5 ] 1.00 - 2.00 sec 1.95 GBytes 16.7 Gbits/sec

[ 5 ] 2.00 - 3.00 sec 1.85 GBytes 15.9 Gbits/sec

[ 5 ] 3.00 - 4.00 sec 1.92 GBytes 16.5 Gbits/sec

[ 5 ] 4.00 - 5.00 sec 2.11 GBytes 18.1 Gbits/sec

[ 5 ] 5.00 - 6.00 sec 1.82 GBytes 15.6 Gbits/sec

[ 5 ] 6.00 - 7.00 sec 2.08 GBytes 17.8 Gbits/sec

[ 5 ] 7.00 - 8.00 sec 2.64 GBytes 22.6 Gbits/sec

[ 5 ] 8.00 - 9.00 sec 2.57 GBytes 22.1 Gbits/sec

[ 5 ] 9.00 - 10.00 sec 1.92 GBytes 16.5 Gbits/sec

[ 5 ] 10.00 - 10.04 sec 74.6 MBytes 15.4 Gbits/sec

In theory yes but cpu=2.4%, other=97.6% and unique snapshot indicate that this was not really a problem, the thread was almost entirely idle as it should be.

This could be because of some earlier slow-running activity on a transport thread on the receiving node. 898985ms is ~15min, seems unlikely to be network congestion.

You don't mention which version you're using, but it looks like it's not a recent 8.x version at least. I'd recommend upgrading, the 8.x series has many performance and stability improvements that target larger clusters like yours.

Hi David,

Thanks for the reply.

  1. Regarding "This could be because of some earlier slow-running activity on a transport thread on the receiving node. 898985ms is ~15min, seems unlikely to be network congestion."

I checked the receiving node(10.246.12.140)'s task list, hot thread and logs, and didn't found anything that would stands out. Any other thing you would suggest us to look further? I have tried jstack as well, but it is hard to find the good time to capture the dump.

  1. Regard the long running tasks "cluster:monitor/nodes/info" and "indices:admin/seq_no/global_checkpoint_sync", any suggestion on the cause? Does both task need to reach out to the other nodes?

  2. Regarding the recommendation for upgrading:
    We are currently using version 7.17.3, although not the latest, but it is supposed to be supported until 9.0.0 according to Elastic Product End of Life Dates | Elastic.

Upgrading the cluster to 8.x is on the roadmap but unfortunately it can not be done in the short time.

Yeah this is the tricky bit indeed. A hot threads or (ideally) jstack dump in the time leading up to the timeout exception is what's needed, but ofc you don't know about that until it's too late. Something like #109204 is going to help here. You could just try running jstack periodically every timeout/2 until you encounter the problem.

Yes. And they're both using the exact same TCP connection, therefore the same Netty event loop.

Supported yes, but it's also been stable (a.k.a. maintenance mode) for over 2 years. Most performance optimizations in this area would be too invasive to backport safely so you're missing out on a lot by not upgrading.

A few stack jstack trace collected on the nodes, some callstack is quite long.



"elasticsearch[datanode-es-jetstream-prd1-gwc6.us-east1-c.c.spcstg1-sedjetstream.internal][transport_worker][T#8]" #70 daemon prio=5 os_prio=0 cpu=488802.10ms elapsed=54081.60s tid=0x00007fe5e4160600 nid=2214543 runnable  [0x00007fe62dffa000]
   java.lang.Thread.State: RUNNABLE
        at java.util.HashMap.hash(java.base@18/HashMap.java:338)
        at java.util.HashMap.put(java.base@18/HashMap.java:610)
        at java.util.HashSet.add(java.base@18/HashSet.java:221)
        at java.util.stream.Collectors$$Lambda$22/0x800000039.accept(java.base@18/Unknown Source)
        at java.util.stream.ReduceOps$3ReducingSink.accept(java.base@18/ReduceOps.java:169)
        at java.util.stream.ReferencePipeline$3$1.accept(java.base@18/ReferencePipeline.java:197)
        at java.util.Spliterators$ArraySpliterator.forEachRemaining(java.base@18/Spliterators.java:992)
        at java.util.stream.AbstractPipeline.copyInto(java.base@18/AbstractPipeline.java:509)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(java.base@18/AbstractPipeline.java:499)
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(java.base@18/ReduceOps.java:921)
        at java.util.stream.AbstractPipeline.evaluate(java.base@18/AbstractPipeline.java:234)
        at java.util.stream.ReferencePipeline.collect(java.base@18/ReferencePipeline.java:682)
        at org.elasticsearch.action.admin.cluster.node.info.NodesInfoRequest$Metric.allMetrics(NodesInfoRequest.java:193)
        at org.elasticsearch.action.admin.cluster.node.info.NodesInfoRequest.all(NodesInfoRequest.java:79)
        at org.elasticsearch.action.admin.cluster.node.info.NodesInfoRequest.<init>(NodesInfoRequest.java:64)
        at org.elasticsearch.rest.action.admin.cluster.RestNodesInfoAction.prepareRequest(RestNodesInfoAction.java:88)
        at org.elasticsearch.rest.action.admin.cluster.RestNodesInfoAction.prepareRequest(RestNodesInfoAction.java:59)
        at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:86)
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$0(SecurityRestFilter.java:90)
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter$$Lambda$6431/0x0000000801b44ed8.accept(Unknown Source)
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)
        at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.lambda$authenticateAndAttachToContext$2(SecondaryAuthenticator.java:84)
        at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator$$Lambda$6434/0x0000000801b45528.accept(Unknown Source)
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)
        at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticate(SecondaryAuthenticator.java:94)
        at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticateAndAttachToContext(SecondaryAuthenticator.java:78)
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$2(SecurityRestFilter.java:85)
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter$$Lambda$6409/0x0000000801b25e78.accept(Unknown Source)
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)
        at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101)
        at org.elasticsearch.action.ActionListener$RunBeforeActionListener.onResponse(ActionListener.java:389)
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.writeAuthToContext(AuthenticatorChain.java:376)
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.finishAuthentication(AuthenticatorChain.java:352)
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.maybeLookupRunAsUser(AuthenticatorChain.java:205)
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$doAuthenticate$1(AuthenticatorChain.java:128)
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain$$Lambda$6155/0x0000000801a64218.accept(Unknown Source)
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)
        at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:141)
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$3(AuthenticatorChain.java:175)
            at org.elasticsearch.xpack.security.authc.AuthenticatorChain$$Lambda$6411/0x0000000801b262a8.accept(Unknown Source)
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)
        at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$3(RealmsAuthenticator.java:219)
        at org.elasticsearch.xpack.security.authc.RealmsAuthenticator$$Lambda$6414/0x0000000801b26900.accept(Unknown Source)
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)
        at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:141)
        at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$0(RealmsAuthenticator.java:164)
        at org.elasticsearch.xpack.security.authc.RealmsAuthenticator$$Lambda$6417/0x0000000801b26d30.accept(Unknown Source)
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$authenticateWithCache$1(CachingUsernamePasswordRealm.java:155)
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm$$Lambda$6453/0x0000000801b45f78.accept(Unknown Source)
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.handleCachedAuthentication(CachingUsernamePasswordRealm.java:242)
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$authenticateWithCache$2(CachingUsernamePasswordRealm.java:139)
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm$$Lambda$6451/0x0000000801b45b48.accept(Unknown Source)
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)
        at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:113)
        at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:55)
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:134)
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:105)
        at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$2(RealmsAuthenticator.java:148)
        at org.elasticsearch.xpack.security.authc.RealmsAuthenticator$$Lambda$6413/0x0000000801b266d8.accept(Unknown Source)
        at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117)
        at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.consumeToken(RealmsAuthenticator.java:233)
        at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.authenticate(RealmsAuthenticator.java:84)
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:171)
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain$$Lambda$6157/0x0000000801a64648.accept(Unknown Source)
        at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135)
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:165)
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain$$Lambda$6157/0x0000000801a64648.accept(Unknown Source)
        at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135)
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:165)
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain$$Lambda$6157/0x0000000801a64648.accept(Unknown Source)
        at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135)
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:165)
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain$$Lambda$6157/0x0000000801a64648.accept(Unknown Source)
        at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117)
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.doAuthenticate(AuthenticatorChain.java:143)
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:104)
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:149)
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:127)
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:79)
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:327)
        at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:393)
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:245)
         at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:382)
        at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:461)
        at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:357)
        at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:35)
        at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:19)
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:48)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1374)
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1237)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1286)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
               at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at java.lang.Thread.run(java.base@18/Thread.java:833)

Another one

elasticsearch[datanode-es-jetstream-prd1-n9v3.us-east1-c.c.spcstg1-sedjetstream.internal][transport_worker][T#12]" #74 daemon prio=5 os_prio=0 cpu=82568366.03ms elapsed=6809612.13s tid=0x00007fcb9429cca0 nid=10644 runnable  [0x00007fcbb15ea000]
   java.lang.Thread.State: RUNNABLE
        at java.util.stream.ReferencePipeline$3$1.accept(java.base@18/ReferencePipeline.java:197)
        at java.util.Spliterators$ArraySpliterator.forEachRemaining(java.base@18/Spliterators.java:992)
        at java.util.stream.AbstractPipeline.copyInto(java.base@18/AbstractPipeline.java:509)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(java.base@18/AbstractPipeline.java:499)
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(java.base@18/ReduceOps.java:921)
        at java.util.stream.AbstractPipeline.evaluate(java.base@18/AbstractPipeline.java:234)
        at java.util.stream.ReferencePipeline.collect(java.base@18/ReferencePipeline.java:682)
        at org.elasticsearch.action.admin.cluster.node.info.NodesInfoRequest$Metric.allMetrics(NodesInfoRequest.java:193)
        at org.elasticsearch.action.admin.cluster.node.info.NodesInfoRequest.addMetrics(NodesInfoRequest.java:106)
        at org.elasticsearch.rest.action.admin.cluster.RestNodesInfoAction.prepareRequest(RestNodesInfoAction.java:97)
        at org.elasticsearch.rest.action.admin.cluster.RestNodesInfoAction.prepareRequest(RestNodesInfoAction.java:59)
        at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:86)
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$0(SecurityRestFilter.java:90)
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter$$Lambda$6451/0x0000000801b78220.accept(Unknown Source)
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)
        at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.lambda$authenticateAndAttachToContext$2(SecondaryAuthenticator.java:84)
        at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator$$Lambda$6455/0x0000000801b78870.accept(Unknown Source)
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)
        at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticate(SecondaryAuthenticator.java:94)
        at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticateAndAttachToContext(SecondaryAuthenticator.java:78)
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$2(SecurityRestFilter.java:85)
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter$$Lambda$6415/0x0000000801b435c0.accept(Unknown Source)
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)
        at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101)
        at org.elasticsearch.action.ActionListener$RunBeforeActionListener.onResponse(ActionListener.java:389)
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.writeAuthToContext(AuthenticatorChain.java:376)
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.finishAuthentication(AuthenticatorChain.java:352)
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.maybeLookupRunAsUser(AuthenticatorChain.java:205)
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$doAuthenticate$1(AuthenticatorChain.java:128)
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain$$Lambda$6131/0x0000000801a96000.accept(Unknown Source)
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)
        at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:141)
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$3(AuthenticatorChain.java:175)
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain$$Lambda$6419/0x0000000801b70000.accept(Unknown Source)
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)
        at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$3(RealmsAuthenticator.java:219)
        at org.elasticsearch.xpack.security.authc.RealmsAuthenticator$$Lambda$6425/0x0000000801b70cb0.accept(Unknown Source)
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)
        at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:141)
        at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$0(RealmsAuthenticator.java:164)
        at org.elasticsearch.xpack.security.authc.RealmsAuthenticator$$Lambda$6430/0x0000000801b71300.accept(Unknown Source)
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$authenticateWithCache$1(CachingUsernamePasswordRealm.java:155)
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm$$Lambda$6449/0x0000000801b73d38.accept(Unknown Source)
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.handleCachedAuthentication(CachingUsernamePasswordRealm.java:242)
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$authenticateWithCache$2(CachingUsernamePasswordRealm.java:139)
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm$$Lambda$6438/0x0000000801b725b8.accept(Unknown Source)
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)
        at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:113)
        at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:55)
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:134)
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:105)
        at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$2(RealmsAuthenticator.java:148)
        at org.elasticsearch.xpack.security.authc.RealmsAuthenticator$$Lambda$6424/0x0000000801b70a88.accept(Unknown Source)
        at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117)
        at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.consumeToken(RealmsAuthenticator.java:233)
        at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.authenticate(RealmsAuthenticator.java:84)
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:171)
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain$$Lambda$6133/0x0000000801a96430.accept(Unknown Source)
        at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135)
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:165)
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain$$Lambda$6133/0x0000000801a96430.accept(Unknown Source)
        at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135)
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:165)
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain$$Lambda$6133/0x0000000801a96430.accept(Unknown Source)
        at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135)
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:165)
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain$$Lambda$6133/0x0000000801a96430.accept(Unknown Source)
        at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117)
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.doAuthenticate(AuthenticatorChain.java:143)
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:104)
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:149)
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:127)
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:79)
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:327)
        at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:393)
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:245)
        at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:382)
        at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:461)
        at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:357)
        at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:35)
        at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:19)
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:48)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1374)
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1237)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1286)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at java.lang.Thread.run(java.base@18/Thread.java:833)
"elasticsearch[master-es-jetstream-prd1-pxmw.us-east1-c.c.spcstg1-sedjetstream.internal][transport_worker][T#7]" #62 daemon prio=5 os_prio=0 cpu=55944949.94ms elapsed=3829243.94s tid=0x00007ffaf0010520 nid=71560 runnable  [0x00007ffb3dc6d000]
   java.lang.Thread.State: RUNNABLE
        at org.elasticsearch.common.bytes.BytesReferenceStreamInput.read(BytesReferenceStreamInput.java:239)
        at org.elasticsearch.common.bytes.BytesReferenceStreamInput.readBytes(BytesReferenceStreamInput.java:208)
        at org.elasticsearch.common.io.stream.FilterStreamInput.readBytes(FilterStreamInput.java:35)
        at org.elasticsearch.common.io.stream.StreamInput.readString(StreamInput.java:481)
        at org.elasticsearch.cluster.node.DiscoveryNode.<init>(DiscoveryNode.java:305)
        at org.elasticsearch.action.support.nodes.BaseNodesRequest$$Lambda$6077/0x0000000801a96508.read(Unknown Source)
        at org.elasticsearch.common.io.stream.StreamInput.readArray(StreamInput.java:1023)
        at org.elasticsearch.common.io.stream.StreamInput.readOptionalArray(StreamInput.java:1029)
        at org.elasticsearch.action.support.nodes.BaseNodesRequest.<init>(BaseNodesRequest.java:46)
        at org.elasticsearch.action.admin.cluster.node.info.NodesInfoRequest.<init>(NodesInfoRequest.java:38)
        at org.elasticsearch.action.admin.cluster.node.info.TransportNodesInfoAction$NodeInfoRequest.<init>(TransportNodesInfoAction.java:102)
        at org.elasticsearch.action.admin.cluster.node.info.TransportNodesInfoAction$$Lambda$5001/0x000000080181ce38.read(Unknown Source)
        at org.elasticsearch.transport.RequestHandlerRegistry.newRequest(RequestHandlerRegistry.java:54)
        at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:229)
        at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:106)
        at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:88)
        at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:743)
        at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler$$Lambda$5891/0x0000000801a06b40.accept(Unknown Source)
        at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:147)
        at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:119)
        at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:84)
        at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:71)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1374)
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1237)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1286)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at java.lang.Thread.run(java.base@18/Thread.java:833)
"elasticsearch[datanode-es-jetstream-prd1-rgkn.us-east1-c.c.spcstg1-sedjetstream.internal][transport_worker][T#15]" #77 daemon prio=5 os_prio=0 cpu=78084769.50ms elapsed=6770261.09s tid=0x00007facd40152f0 nid=11010 runnable  [0x00007facbf9fb000]
   java.lang.Thread.State: RUNNABLE
        at org.elasticsearch.common.io.stream.StreamOutput.write(StreamOutput.java:516)
        at org.elasticsearch.common.transport.TransportAddress.writeTo(TransportAddress.java:75)
        at org.elasticsearch.cluster.node.DiscoveryNode.writeTo(DiscoveryNode.java:365)
        at org.elasticsearch.common.io.stream.StreamOutput.lambda$writeOptionalArray$33(StreamOutput.java:972)
        at org.elasticsearch.common.io.stream.StreamOutput$$Lambda$6494/0x0000000801b9d730.write(Unknown Source)
        at org.elasticsearch.common.io.stream.StreamOutput.writeArray(StreamOutput.java:941)
        at org.elasticsearch.common.io.stream.StreamOutput.writeOptionalArray(StreamOutput.java:954)
        at org.elasticsearch.common.io.stream.StreamOutput.writeOptionalArray(StreamOutput.java:972)
        at org.elasticsearch.action.support.nodes.BaseNodesRequest.writeTo(BaseNodesRequest.java:102)
        at org.elasticsearch.action.admin.cluster.node.info.NodesInfoRequest.writeTo(NodesInfoRequest.java:141)
        at org.elasticsearch.action.admin.cluster.node.info.TransportNodesInfoAction$NodeInfoRequest.writeTo(TransportNodesInfoAction.java:112)
        at org.elasticsearch.transport.OutboundMessage.serialize(OutboundMessage.java:75)
        at org.elasticsearch.transport.OutboundHandler.sendMessage(OutboundHandler.java:180)
        at org.elasticsearch.transport.OutboundHandler.sendRequest(OutboundHandler.java:109)
        at org.elasticsearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:288)
        at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:975)
        at org.elasticsearch.transport.TransportService$$Lambda$4399/0x000000080164e700.sendRequest(Unknown Source)
        at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.sendWithUser(SecurityServerTransportInterceptor.java:206)
        at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.access$300(SecurityServerTransportInterceptor.java:53)
        at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$1.sendRequest(SecurityServerTransportInterceptor.java:167)
        at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:874)
        at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:797)
        at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.start(TransportNodesAction.java:246)
        at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:123)
        at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:40)
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:179)
        at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53)
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177)
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:190)
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter$$Lambda$6171/0x0000000801ab7708.accept(Unknown Source)
        at org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:219)
        at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$4(AuthorizationService.java:391)
        at org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$6135/0x0000000801a9b5f0.accept(Unknown Source)
        at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:967)
        at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:931)
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)
        at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$5(AuthorizationService.java:405)
        at org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$6137/0x0000000801a9ba30.accept(Unknown Source)
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)
        at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeClusterAction(RBACEngine.java:168)
        at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:395)
        at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:371)
        at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:256)
        at org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$6131/0x0000000801a9a8f8.accept(Unknown Source)
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)
        at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:138)
        at org.elasticsearch.xpack.security.authz.RBACEngine$$Lambda$6133/0x0000000801a9ad28.accept(Unknown Source)
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:277)
        at org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:144)
        at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:127)
        at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:258)
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:186)
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter$$Lambda$6163/0x0000000801ab6000.accept(Unknown Source)
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)
        at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101)
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:102)
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171)
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:182)
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:124)
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177)
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:154)
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:82)
        at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:95)
        at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:73)
        at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:407)
        at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:708)
        at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.nodesInfo(AbstractClient.java:791)
        at org.elasticsearch.rest.action.admin.cluster.RestNodesInfoAction.lambda$prepareRequest$0(RestNodesInfoAction.java:62)
        at org.elasticsearch.rest.action.admin.cluster.RestNodesInfoAction$$Lambda$6493/0x0000000801b9d298.accept(Unknown Source)
        at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:109)
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$0(SecurityRestFilter.java:90)
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter$$Lambda$6460/0x0000000801b8f780.accept(Unknown Source)
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)
        at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.lambda$authenticateAndAttachToContext$2(SecondaryAuthenticator.java:84)
        at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator$$Lambda$6463/0x0000000801b8fdd0.accept(Unknown Source)
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)
        at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticate(SecondaryAuthenticator.java:94)
        at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticateAndAttachToContext(SecondaryAuthenticator.java:78)
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$2(SecurityRestFilter.java:85)
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter$$Lambda$6438/0x0000000801b66968.accept(Unknown Source)
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)
        at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101)
        at org.elasticsearch.action.ActionListener$RunBeforeActionListener.onResponse(ActionListener.java:389)
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.writeAuthToContext(AuthenticatorChain.java:376)
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.finishAuthentication(AuthenticatorChain.java:352)
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.maybeLookupRunAsUser(AuthenticatorChain.java:205)
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$doAuthenticate$1(AuthenticatorChain.java:128)
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain$$Lambda$6166/0x0000000801ab6858.accept(Unknown Source)
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)
        at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:141)
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$3(AuthenticatorChain.java:175)
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain$$Lambda$6440/0x0000000801b66d98.accept(Unknown Source)
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)
        at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$3(RealmsAuthenticator.java:219)
        at org.elasticsearch.xpack.security.authc.RealmsAuthenticator$$Lambda$6443/0x0000000801b673f0.accept(Unknown Source)
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)
        at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:141)
        at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$0(RealmsAuthenticator.java:164)
        at org.elasticsearch.xpack.security.authc.RealmsAuthenticator$$Lambda$6446/0x0000000801b67820.accept(Unknown Source)
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$authenticateWithCache$1(CachingUsernamePasswordRealm.java:155)
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm$$Lambda$6479/0x0000000801b94840.accept(Unknown Source)
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.handleCachedAuthentication(CachingUsernamePasswordRealm.java:242)
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$authenticateWithCache$2(CachingUsernamePasswordRealm.java:139)
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm$$Lambda$6477/0x0000000801b94410.accept(Unknown Source)
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)
...

None of this stuff looks particularly long-running and shouldn't cause a problem unless you're running hundreds of nodes-info requests at once. Nonetheless, this is all rather different in recent 8.x versions.