Stack Monitoring not showing data node statistics

MakoWish · February 16, 2020, 2:04am

For some reason, Stack Monitoring is now showing "Total shards: 0", "Documents: 0", "Data: 0.0B", and the node count is not including my data nodes. Why is no data present for any of my data nodes? If I restart Elasticsearch on one of the data nodes, it will briefly increase the node count by 1, but it then returns back to not including that node. I have seen a few suggestions to check the time on the servers, and they are all correct. Not sure where else to look to figure this out.

Elasticsearch 7.6.0
Kibana 7.6.0

GET _cluster/health?pretty
{
  "cluster_name" : "Cluster-Name",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 9,
  "number_of_data_nodes" : 5,
  "active_primary_shards" : 360,
  "active_shards" : 720,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

GET _nodes/stats
{
  "_nodes" : {
    "total" : 9,
    "successful" : 4,
    "failed" : 5,
    "failures" : [
      {
        "type" : "failed_node_exception",
        "reason" : "Failed node [NzA6DLQoSvyYPZpC1_nc_w]",
        "node_id" : "NzA6DLQoSvyYPZpC1_nc_w",
        "caused_by" : {
          "type" : "illegal_state_exception",
          "reason" : "Negative longs unsupported, use writeLong or writeZLong for negative numbers [-410]"
        }
      },
      {
        "type" : "failed_node_exception",
        "reason" : "Failed node [vzLH-Rc3RtqbFVnh6j39Cw]",
        "node_id" : "vzLH-Rc3RtqbFVnh6j39Cw",
        "caused_by" : {
          "type" : "illegal_state_exception",
          "reason" : "Negative longs unsupported, use writeLong or writeZLong for negative numbers [-2449]"
        }
      },
      {
        "type" : "failed_node_exception",
        "reason" : "Failed node [mVi86ywZQ8e8cWJ2ype9cQ]",
        "node_id" : "mVi86ywZQ8e8cWJ2ype9cQ",
        "caused_by" : {
          "type" : "illegal_state_exception",
          "reason" : "Negative longs unsupported, use writeLong or writeZLong for negative numbers [-116]"
        }
      },
      {
        "type" : "failed_node_exception",
        "reason" : "Failed node [B9I4onxkSRCHnQsl_VdX2Q]",
        "node_id" : "B9I4onxkSRCHnQsl_VdX2Q",
        "caused_by" : {
          "type" : "illegal_state_exception",
          "reason" : "Negative longs unsupported, use writeLong or writeZLong for negative numbers [-62]"
        }
      },
      {
        "type" : "failed_node_exception",
        "reason" : "Failed node [rnj8pp48S1mX2fNej_TM-w]",
        "node_id" : "rnj8pp48S1mX2fNej_TM-w",
        "caused_by" : {
          "type" : "illegal_state_exception",
          "reason" : "Negative longs unsupported, use writeLong or writeZLong for negative numbers [-227]"
        }
      }
    ]
  },
  "cluster_name" : "Cluster-Name",
  "nodes" : {
    THE FOUR UP NODES
  }
}

When I look in Kibana, it shows 4 nodes (3x master and 1x coordinator), and no counts for shards, nodes, data, etc.. All the nodes do appear listed properly as up on the bottom of the page.

MakoWish · February 16, 2020, 5:01am

Looks like this is the source of the error in the _node/stats message. Found this in the elasticsearch.server logs. Issue with authentication? </ scratching_head>

send message failed [channel: Netty4TcpChannel{localAddress=/<DataNode5>:9300, remoteAddress=/<MasterNode3>:50402}]
java.lang.IllegalStateException: Negative longs unsupported, use writeLong or writeZLong for negative numbers [-42]
	at org.elasticsearch.common.io.stream.StreamOutput.writeVLong(StreamOutput.java:299) ~[elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.ingest.IngestStats$Stats.writeTo(IngestStats.java:197) ~[elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.ingest.IngestStats.writeTo(IngestStats.java:103) ~[elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.common.io.stream.StreamOutput.writeOptionalWriteable(StreamOutput.java:897) ~[elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.action.admin.cluster.node.stats.NodeStats.writeTo(NodeStats.java:255) ~[elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.transport.OutboundMessage.writeMessage(OutboundMessage.java:87) ~[elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.transport.OutboundMessage.serialize(OutboundMessage.java:64) ~[elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.transport.OutboundHandler$MessageSerializer.get(OutboundHandler.java:166) ~[elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.transport.OutboundHandler$MessageSerializer.get(OutboundHandler.java:152) ~[elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:199) [elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.transport.OutboundHandler.internalSend(OutboundHandler.java:129) [elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.transport.OutboundHandler.sendMessage(OutboundHandler.java:124) [elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.transport.OutboundHandler.sendResponse(OutboundHandler.java:104) [elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.transport.TcpTransportChannel.sendResponse(TcpTransportChannel.java:64) [elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:54) [elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:244) [elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:240) [elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:257) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225) [elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$0(SecurityServerTransportInterceptor.java:306) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$4(AuthorizationService.java:241) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:616) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:591) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) [elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeClusterAction(RBACEngine.java:144) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:243) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:227) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:193) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) [elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:117) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:208) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:123) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:111) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:195) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$1(ServerTransportFilter.java:130) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:248) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:310) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:321) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:245) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:196) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:139) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:121) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:313) [x-pack-security-7.6.0.jar:7.6.0]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:264) [elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.0.jar:7.6.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:830) [?:?]

MakoWish · February 16, 2020, 5:50am

Looks to possibly be related to https://github.com/elastic/elasticsearch/issues/52339

system · March 15, 2020, 5:50am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Monitoring -> Elasticsearch -> Nodes - no data nodes( Kibana elastic-stack-monitoring	22	2980	March 7, 2019
All Data nodes are offline except master node in Stack monitoring Kibana	2	818	February 20, 2020
Kibana display zero shards in Monitoring nodes tab Elasticsearch elastic-stack-monitoring	16	2366	September 27, 2019
All nodes except Master show as “Offline” Kibana elastic-stack-monitoring	12	1667	February 20, 2020
Nodes fail to list Kibana elastic-stack-monitoring	7	341	September 5, 2019

Stack Monitoring not showing data node statistics

Related topics