Real cluster state size

Elastic Stack Elasticsearch
1

When I try to get the cluster state size via GET _nodes/stats in kibana dev tools it gives me this data from the node, which is holding cluster state now

"serialized_cluster_states": {
  "full_states": {
    "count": 772,
    "uncompressed_size_in_bytes": 11241694795,
    "compressed_size_in_bytes": 4222615218
  },
  "diffs": {
    "count": 2187988,
    "uncompressed_size_in_bytes": 33665369857,
    "compressed_size_in_bytes": 8659386985
  }
},

but when I check it on any node using command
curl -X GET "https://<address_here>/_cluster/state?pretty" -k -u "<creds_here>" -o cluster_state.json
it gives me cluster_state.json with size of 1013867688 ~ 1GB
and if try to take it without pretty it will be 323170792 ~ 323MB

So this value is nowhere near 11GB that is in serialized_cluster_states

How can I get the actual size of cluster state?

2

This is the total size of all full cluster states the node has serialized since it started. So on average that’s a little under 14MiB per state:

$ echo 'scale=2; 11241694795/772/1024^2' | bc
13.88

What do you mean by “actual size”?

3

Oh, I think I get it

So 11241694795 is the full state from start

And the current cluster state is the state from /_cluster/state and in my situation It is around 320MB?

By "actual size" I meant the current one, that the cluster is holding

Is there any documentation, where I can read about uncompressed_size_in_bytes? cause I didn't find one

4

Sure but in what sense? The size on the wire between nodes is the compressed size (average ~5.2MiB according to your numbers). On disk it's stored as a collection of files in ${path.data}/_state with a different format so I expect it's a little larger. Rendered as JSON it's much larger than either of these. Which of these is the "actual size" in the sense in which you're interested?

I don't think this is documented, no, it's only really useful for some fairly deep troubleshooting when read alongside the code itself, it doesn't have much meaning otherwise.

5

In a matter of size I am interested in the current size of the cluster state, which is located in ${path.data}/_state, but in a matter of content (which is the thing that I am originally looking at and trying to find what's in there currently) I think the second one

got it

6

And if we are talking about the cluster size itself - is there any documentation (not the unofficial one), where I can find what things can affect the cluster state and is there any way to find out the normal cluster state size (maybe calculate it somehow)?

7

I’m still not really sure what you’re asking here, but a cluster state that renders as ~300MiB of JSON is not particularly unusual or concerning.

8

I am just trying to dive deeper into elastic cluster state

okey, got it

Thank you for your time and for answering my questions))

1 Like