Cluster Stats Total On Disk Storage Used


(Mike) #1

I'm trying to figure out the total on disk storage used by my cluster. The cluster stats api provides the indices store size_in_bytes as well as nodes fs total and free metrics. I think I'm more concerned with the nodes file system metrics (fs) so in my case I'd take the file system total and subtract the available to get the total disk space being used. Does that sound right? So what is the indices store size_in_bytes metric?

{
timestamp: 1454948431251,
cluster_name: "escluster",
status: "green",
indices: {
count: 21,
shards: {
total: 1122,
primaries: 194,
replication: 4.783505154639175,
index: {
shards: {
min: 3,
max: 60,
avg: 53.42857142857143
},
primaries: {
min: 1,
max: 10,
avg: 9.238095238095237
},
replication: {
min: 2,
max: 5,
avg: 4.571428571428571
}
}
},
docs: {
count: 468870065,
deleted: 125997931
},
store: {
size: "1.8tb",
size_in_bytes: 1990473179532,
throttle_time: "9.8d",
throttle_time_in_millis: 853595965
},
fielddata: {
memory_size: "11.7gb",
memory_size_in_bytes: 12661159396,
evictions: 0
},
filter_cache: {
memory_size: "5.4gb",
memory_size_in_bytes: 5807821648,
evictions: 0
},
id_cache: {
memory_size: "0b",
memory_size_in_bytes: 0
},
completion: {
size: "0b",
size_in_bytes: 0
},
segments: {
count: 16096,
memory: "19.1gb",
memory_in_bytes: 20578612072,
index_writer_memory: "23.2mb",
index_writer_memory_in_bytes: 24406829,
index_writer_max_memory: "80.7gb",
index_writer_max_memory_in_bytes: 86663689921,
version_map_memory: "2.7mb",
version_map_memory_in_bytes: 2850952,
fixed_bit_set: "4.5gb",
fixed_bit_set_memory_in_bytes: 4845908496
},
percolate: {
total: 0,
get_time: "0s",
time_in_millis: 0,
current: 0,
memory_size_in_bytes: -1,
memory_size: "-1b",
queries: 0
}
},
nodes: {
count: {
total: 17,
master_only: 3,
data_only: 14,
master_data: 0,
client: 0
},
versions: [
"1.7.2"
],
os: {
available_processors: 720,
mem: {
total: "4.1tb",
total_in_bytes: 4569933668352
},
cpu: [
{
vendor: "Intel",
model: "Xeon",
mhz: 2500,
total_cores: 40,
total_sockets: 1,
cores_per_socket: 32,
cache_size: "25kb",
cache_size_in_bytes: 25600,
count: 17
}
]
},
process: {
cpu: {
percent: 1772
},
open_file_descriptors: {
min: 1180,
max: 4698,
avg: 3987
}
},
jvm: {
max_uptime: "94.9d",
max_uptime_in_millis: 8201602618,
versions: [
{
version: "1.8.0_40",
vm_name: "Java HotSpot(TM) 64-Bit Server VM",
vm_version: "25.40-b25",
vm_vendor: "Oracle Corporation",
count: 17
}
],
mem: {
heap_used: "151.8gb",
heap_used_in_bytes: 163032033888,
heap_max: "428.2gb",
heap_max_in_bytes: 459789631488
},
threads: 15100
},
fs: {
total: "23.3tb",
total_in_bytes: 25668544499712,
free: "16.3tb",
free_in_bytes: 18019465801728,
available: "15.2tb",
available_in_bytes: 16715538493440,
disk_reads: 3651846331,
disk_writes: 683952779384,
disk_io_op: 687604625715,
disk_read_size: "81.4tb",
disk_read_size_in_bytes: 89555096878080,
disk_write_size: "2.4pb",
disk_write_size_in_bytes: 2800925002510336,
disk_io_size: "2.5pb",
disk_io_size_in_bytes: 2890480099388416,
disk_queue: "0",
disk_service_time: "1"
},
plugins: [
{
name: "head",
version: "NA",
description: "No description found.",
url: "/_plugin/head/",
jvm: false,
site: true
},
{
name: "bigdesk",
version: "NA",
description: "No description found.",
url: "/_plugin/bigdesk/",
jvm: false,
site: true
},
{
name: "marvel",
version: "1.3.0",
description: "Elasticsearch Management & Monitoring",
url: "/_plugin/marvel/",
jvm: true,
site: true
},
{
name: "reindex",
version: "NA",
description: "ReIndex Plugin",
jvm: true,
site: false
},
{
name: "HQ",
version: "NA",
description: "No description found.",
url: "/_plugin/HQ/",
jvm: false,
site: true
}
]
}
}


(Ali Beyad) #2

That's correct, the total minus the free space. The free space is the amount of free space on the disk, whereas the available space is the amount of that disk space available to the JVM.

The size_in_bytes for the indices store is the size of the actual Lucene on-disk segments that represents the search index, which is but part of all total storage Elasticsearch uses to accomplish searches / aggregations.


(system) #3