Stack Monitoring not showing data node statistics

For some reason, Stack Monitoring is now showing "Total shards: 0", "Documents: 0", "Data: 0.0B", and the node count is not including my data nodes. Why is no data present for any of my data nodes? If I restart Elasticsearch on one of the data nodes, it will briefly increase the node count by 1, but it then returns back to not including that node. I have seen a few suggestions to check the time on the servers, and they are all correct. Not sure where else to look to figure this out.

Elasticsearch 7.6.0
Kibana 7.6.0

GET _cluster/health?pretty
{
  "cluster_name" : "Cluster-Name",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 9,
  "number_of_data_nodes" : 5,
  "active_primary_shards" : 360,
  "active_shards" : 720,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

GET _nodes/stats
{
  "_nodes" : {
    "total" : 9,
    "successful" : 4,
    "failed" : 5,
    "failures" : [
      {
        "type" : "failed_node_exception",
        "reason" : "Failed node [NzA6DLQoSvyYPZpC1_nc_w]",
        "node_id" : "NzA6DLQoSvyYPZpC1_nc_w",
        "caused_by" : {
          "type" : "illegal_state_exception",
          "reason" : "Negative longs unsupported, use writeLong or writeZLong for negative numbers [-410]"
        }
      },
      {
        "type" : "failed_node_exception",
        "reason" : "Failed node [vzLH-Rc3RtqbFVnh6j39Cw]",
        "node_id" : "vzLH-Rc3RtqbFVnh6j39Cw",
        "caused_by" : {
          "type" : "illegal_state_exception",
          "reason" : "Negative longs unsupported, use writeLong or writeZLong for negative numbers [-2449]"
        }
      },
      {
        "type" : "failed_node_exception",
        "reason" : "Failed node [mVi86ywZQ8e8cWJ2ype9cQ]",
        "node_id" : "mVi86ywZQ8e8cWJ2ype9cQ",
        "caused_by" : {
          "type" : "illegal_state_exception",
          "reason" : "Negative longs unsupported, use writeLong or writeZLong for negative numbers [-116]"
        }
      },
      {
        "type" : "failed_node_exception",
        "reason" : "Failed node [B9I4onxkSRCHnQsl_VdX2Q]",
        "node_id" : "B9I4onxkSRCHnQsl_VdX2Q",
        "caused_by" : {
          "type" : "illegal_state_exception",
          "reason" : "Negative longs unsupported, use writeLong or writeZLong for negative numbers [-62]"
        }
      },
      {
        "type" : "failed_node_exception",
        "reason" : "Failed node [rnj8pp48S1mX2fNej_TM-w]",
        "node_id" : "rnj8pp48S1mX2fNej_TM-w",
        "caused_by" : {
          "type" : "illegal_state_exception",
          "reason" : "Negative longs unsupported, use writeLong or writeZLong for negative numbers [-227]"
        }
      }
    ]
  },
  "cluster_name" : "Cluster-Name",
  "nodes" : {
    THE FOUR UP NODES
  }
}

When I look in Kibana, it shows 4 nodes (3x master and 1x coordinator), and no counts for shards, nodes, data, etc.. All the nodes do appear listed properly as up on the bottom of the page.

Looks like this is the source of the error in the _node/stats message. Found this in the elasticsearch.server logs. Issue with authentication? </ scratching_head>

send message failed [channel: Netty4TcpChannel{localAddress=/<DataNode5>:9300, remoteAddress=/<MasterNode3>:50402}]
java.lang.IllegalStateException: Negative longs unsupported, use writeLong or writeZLong for negative numbers [-42]
	at org.elasticsearch.common.io.stream.StreamOutput.writeVLong(StreamOutput.java:299) ~[elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.ingest.IngestStats$Stats.writeTo(IngestStats.java:197) ~[elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.ingest.IngestStats.writeTo(IngestStats.java:103) ~[elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.common.io.stream.StreamOutput.writeOptionalWriteable(StreamOutput.java:897) ~[elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.action.admin.cluster.node.stats.NodeStats.writeTo(NodeStats.java:255) ~[elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.transport.OutboundMessage.writeMessage(OutboundMessage.java:87) ~[elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.transport.OutboundMessage.serialize(OutboundMessage.java:64) ~[elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.transport.OutboundHandler$MessageSerializer.get(OutboundHandler.java:166) ~[elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.transport.OutboundHandler$MessageSerializer.get(OutboundHandler.java:152) ~[elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:199) [elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.transport.OutboundHandler.internalSend(OutboundHandler.java:129) [elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.transport.OutboundHandler.sendMessage(OutboundHandler.java:124) [elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.transport.OutboundHandler.sendResponse(OutboundHandler.java:104) [elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.transport.TcpTransportChannel.sendResponse(TcpTransportChannel.java:64) [elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:54) [elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:244) [elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:240) [elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:257) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225) [elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$0(SecurityServerTransportInterceptor.java:306) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$4(AuthorizationService.java:241) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:616) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:591) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) [elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeClusterAction(RBACEngine.java:144) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:243) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:227) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:193) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) [elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:117) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:208) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:123) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:111) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:195) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$1(ServerTransportFilter.java:130) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:248) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:310) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:321) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:245) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:196) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:139) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:121) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:313) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:264) [elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.0.jar:7.6.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:830) [?:?]

Looks to possibly be related to https://github.com/elastic/elasticsearch/issues/52339

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.