ES Cluster Stats API Returning False Values

Elastic Stack Elasticsearch
1

I run some cronjobs that use the cluster stats api to retrieve the total cluster space and the current available cluster space in bytes and run some calculations on them. However, one of our clusters is now returning false values for those categories.

Specifically, this api call:

curl -XGET "http://$(hostname -i):9200/_cluster/stats?human&pretty"

is returning these values (extracted just the fs values since those are the ones we use in calculations)

"fs" : {
  "total" : "8192pb",
  "total_in_bytes" : 9223372036854775807,
  "free" : "8192pb",
  "free_in_bytes" : 9223372036854775807,
  "available" : "8192pb",
  "available_in_bytes" : 9223372036854775807
}

The cluster ACTUALLY consists of 3 nodes with 800gb each so ~2.5tb in total. Not pb's.

Currently, I have even gone in and manually deleted some indices, but these fs values seem to be stuck exactly like this.

Any tips on where to start fixing this would be greatly appreciated!!

Thank you!

2

What version are you on? What OS?

3

Ubuntu 14.04

Over night our fs stats seem to have returned to normal. Very weird issue though, I'd be interested to hear if anyone else has experienced anything similar.

4

Update, this issue still exists. I had believed that it had resolved itself because I queried the cluster on another node and got back accurate values. However, it has become clear that this error is only happening on a single node.

These stats appear to be gathered locally then - what could be corrupted that could cause a result like this??

5

What version of ES is this?
How did you install it? Are you using virtualisation or containerisation?

6

These are running on virtual machines in GCS and were installed via chef by unpacking the tarball made available at https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.4.0.tar.gz.

We have noticed these numbers are also showing up in our kibana monitoring interface:

Where 8EB => 8 Exabytes, but when I go into the nodes page I can see the actual values

image.png1194×61 8.49 KB

7

@pickypg any thoughts here?

8

The nodes listing and the cluster summary come from two different APIs:

The node stats are literally the _nodes/stats API from Elasticsearch, per node, while the cluster summary is the _cluster/stats endpoint.

Something is weird making the _cluster/stats report so much data. I noticed in the picture that it says you have 4 nodes, but only 3 nodes appear? What is the 4th node?

9

The 4th node is a tribe to allow this kibana instance to query across several of our other clusters.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.