Marvel reporting incorrect free disk space?

Tony_Su · January 30, 2014, 10:39pm

Deployed in a VMware environment.
Data is being consumed and is successful from source to ES

source > logstash shipper(and parser) > redis > logstash > es cluster

ES Cluster
All openSUSE 13.1
Machine 1 - 4GB ram - 16GB HDD - openSUSE/LXDE -ES -Logstash - netcat
Machine 2 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
Machine 3 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
Machine 4 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
Machine 5 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES

Marvel reports the disk space likely correctly only on the initial machine
(8.8GB free)
For machines 2-5, free disk space is reported as 434.3 GB

For at least the next day, I'm going to be trying to interpret my results
based on the existing Marvel documentation, but this anomaly jumped out at
me.
I don't know how the disk free space is read by Marvel, maybe it's using a
method that doesn't understand a VMware virtual disk? Is pretty ordinary,
formatted ext4. Maybe a system level dependency is missing on these
machines?

Thx,
Tony

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1904b915-a5a6-4712-b6f7-9635755d1ff9%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Boaz_Leskes · January 31, 2014, 8:52am

Hi Tony,

I suspect this is an issue with the reporting of the Nodes Stats API (based
on sigar). Can you please post the output of calling:

curl -XGET "http://localhost:9200/_cluster/nodes/_local/stats?clear&fs"

&

df -h

on one of the machines 2-5?

Thx,
Boaz

On Thursday, January 30, 2014 11:39:48 PM UTC+1, Tony Su wrote:

Deployed in a VMware environment.
Data is being consumed and is successful from source to ES

source > logstash shipper(and parser) > redis > logstash > es cluster

ES Cluster
All openSUSE 13.1
Machine 1 - 4GB ram - 16GB HDD - openSUSE/LXDE -ES -Logstash - netcat
Machine 2 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
Machine 3 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
Machine 4 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
Machine 5 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES

Marvel reports the disk space likely correctly only on the initial machine
(8.8GB free)
For machines 2-5, free disk space is reported as 434.3 GB

For at least the next day, I'm going to be trying to interpret my results
based on the existing Marvel documentation, but this anomaly jumped out at
me.
I don't know how the disk free space is read by Marvel, maybe it's using a
method that doesn't understand a VMware virtual disk? Is pretty ordinary,
formatted ext4. Maybe a system level dependency is missing on these
machines?

Thx,
Tony

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/28c0440b-54eb-4591-9c15-fe2b5f253632%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Tony_Su · January 31, 2014, 4:10pm

SIGA is great if that is what you're using...
If you'd like I can also test installing that separately to see if it makes
a diff...
Tony

curl -XGET "http://localhost:9200/_cluster/nodes/_local/stats?clear&fs"

{"cluster_name":"elasticsearch","nodes":{"8f-ZbebNT2e23DArE6WnIw":{"timestamp":1391183667580,"name":"ELASTICSEARCH-4","transport_address":"inet[192.168.248.172/192.168.248.172:9300]","indices":{"docs":{"count":2180,"deleted":0},"store":{"size":"1.5mb","size_in_bytes":1631986,"throttle_time":"0s","throttle_time_in_millis":0},"indexing":{"index_total":0,"index_time":"0s","index_time_in_millis":0,"index_current":0,"delete_total":0,"delete_time":"0s","delete_time_in_millis":0,"delete_current":0},"get":{"total":0,"get_time":"0s","time_in_millis":0,"exists_total":0,"exists_time":"0s","exists_time_in_millis":0,"missing_total":0,"missing_time":"0s","missing_time_in_millis":0,"current":0},"search":{"open_contexts":0,"query_total":0,"query_time":"0s","query_time_in_millis":0,"query_current":0,"fetch_total":0,"fetch_time":"0s","fetch_time_in_millis":0,"fetch_current":0}}}}}

df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 15G 2.4G 12G 17% /
devtmpfs 743M 32K 743M 1% /dev
tmpfs 748M 0 748M 0% /dev/shm
tmpfs 748M 3.1M 744M 1% /run
tmpfs 748M 0 748M 0% /sys/fs/cgroup
tmpfs 748M 3.1M 744M 1% /var/run
tmpfs 748M 3.1M 744M 1% /var/lock
.host:/ 905G 467G 438G 52% /mnt/hgfs

On Friday, January 31, 2014 12:52:21 AM UTC-8, Boaz Leskes wrote:

Hi Tony,

I suspect this is an issue with the reporting of the Nodes Stats API
(based on sigar). Can you please post the output of calling:

curl -XGET "http://localhost:9200/_cluster/nodes/_local/stats?clear&fs"

&

df -h

on one of the machines 2-5?

Thx,
Boaz

On Thursday, January 30, 2014 11:39:48 PM UTC+1, Tony Su wrote:

Deployed in a VMware environment.
Data is being consumed and is successful from source to ES

source > logstash shipper(and parser) > redis > logstash > es cluster

ES Cluster
All openSUSE 13.1
Machine 1 - 4GB ram - 16GB HDD - openSUSE/LXDE -ES -Logstash - netcat
Machine 2 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
Machine 3 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
Machine 4 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
Machine 5 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES

Marvel reports the disk space likely correctly only on the initial
machine (8.8GB free)
For machines 2-5, free disk space is reported as 434.3 GB

For at least the next day, I'm going to be trying to interpret my results
based on the existing Marvel documentation, but this anomaly jumped out at
me.
I don't know how the disk free space is read by Marvel, maybe it's using
a method that doesn't understand a VMware virtual disk? Is pretty ordinary,
formatted ext4. Maybe a system level dependency is missing on these
machines?

Thx,
Tony

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5d930324-3d15-4a30-9f7c-c70100d201de%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Boaz_Leskes · January 31, 2014, 4:42pm

Thx.

Something went wrong with the curl command as it returned indices stats
only (which is the default). The "&fs" should have caused it to return file
system stats. Perhaps something went wrong with the '&'. Can you double
check?

On Friday, January 31, 2014 5:10:02 PM UTC+1, Tony Su wrote:

SIGA is great if that is what you're using...
If you'd like I can also test installing that separately to see if it
makes a diff...
Tony

curl -XGET "http://localhost:9200/_cluster/nodes/_local/stats?clear&fs"

{"cluster_name":"elasticsearch","nodes":{"8f-ZbebNT2e23DArE6WnIw":{"timestamp":1391183667580,"name":"ELASTICSEARCH-4","transport_address":"inet[
192.168.248.172/192.168.248.172:9300
]","indices":{"docs":{"count":2180,"deleted":0},"store":{"size":"1.5mb","size_in_bytes":1631986,"throttle_time":"0s","throttle_time_in_millis":0},"indexing":{"index_total":0,"index_time":"0s","index_time_in_millis":0,"index_current":0,"delete_total":0,"delete_time":"0s","delete_time_in_millis":0,"delete_current":0},"get":{"total":0,"get_time":"0s","time_in_millis":0,"exists_total":0,"exists_time":"0s","exists_time_in_millis":0,"missing_total":0,"missing_time":"0s","missing_time_in_millis":0,"current":0},"search":{"open_contexts":0,"query_total":0,"query_time":"0s","query_time_in_millis":0,"query_current":0,"fetch_total":0,"fetch_time":"0s","fetch_time_in_millis":0,"fetch_current":0}}}}}

df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 15G 2.4G 12G 17% /
devtmpfs 743M 32K 743M 1% /dev
tmpfs 748M 0 748M 0% /dev/shm
tmpfs 748M 3.1M 744M 1% /run
tmpfs 748M 0 748M 0% /sys/fs/cgroup
tmpfs 748M 3.1M 744M 1% /var/run
tmpfs 748M 3.1M 744M 1% /var/lock
.host:/ 905G 467G 438G 52% /mnt/hgfs

On Friday, January 31, 2014 12:52:21 AM UTC-8, Boaz Leskes wrote:

Hi Tony,

I suspect this is an issue with the reporting of the Nodes Stats API
(based on sigar). Can you please post the output of calling:

curl -XGET "http://localhost:9200/_cluster/nodes/_local/stats?clear&fs"

&

df -h

on one of the machines 2-5?

Thx,
Boaz

On Thursday, January 30, 2014 11:39:48 PM UTC+1, Tony Su wrote:

Deployed in a VMware environment.
Data is being consumed and is successful from source to ES

source > logstash shipper(and parser) > redis > logstash > es cluster

ES Cluster
All openSUSE 13.1
Machine 1 - 4GB ram - 16GB HDD - openSUSE/LXDE -ES -Logstash - netcat
Machine 2 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
Machine 3 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
Machine 4 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
Machine 5 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES

Marvel reports the disk space likely correctly only on the initial
machine (8.8GB free)
For machines 2-5, free disk space is reported as 434.3 GB

For at least the next day, I'm going to be trying to interpret my
results based on the existing Marvel documentation, but this anomaly jumped
out at me.
I don't know how the disk free space is read by Marvel, maybe it's using
a method that doesn't understand a VMware virtual disk? Is pretty ordinary,
formatted ext4. Maybe a system level dependency is missing on these
machines?

Thx,
Tony

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fcd17f7e-13b8-4f88-ad63-76dde148975f%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Tony_Su · January 31, 2014, 4:55pm

Sorry, you're right. Typo in my curl command.

corrected

{"cluster_name":"elasticsearch","nodes":{"8f-ZbebNT2e23DArE6WnIw":{"timestamp":1391187042684,"name":"ELASTICSEARCH-4","transport_address":"inet[192.168.248.172/192.168.248.172:9300]","fs":{"timestamp":1391187042685,"total":{"total":"904.3gb","total_in_bytes":971057917952,"free":"431.7gb","free_in_bytes":463607828480,"available":"431.7gb","available_in_bytes":463607828480},"data":[{"path":"/mnt/hgfs/DataShared/elasticsearch/nodes/0","mount":"/mnt/hgfs","dev":".host:/","total":"904.3gb","total_in_bytes":971057917952,"free":"431.7gb","free_in_bytes":463607828480,"available":"431.7gb","available_in_bytes":463607828480}]}}}}

On Friday, January 31, 2014 8:42:02 AM UTC-8, Boaz Leskes wrote:

Thx.

Something went wrong with the curl command as it returned indices stats
only (which is the default). The "&fs" should have caused it to return file
system stats. Perhaps something went wrong with the '&'. Can you double
check?

On Friday, January 31, 2014 5:10:02 PM UTC+1, Tony Su wrote:

SIGA is great if that is what you're using...
If you'd like I can also test installing that separately to see if it
makes a diff...
Tony

curl -XGET "http://localhost:9200/_cluster/nodes/_local/stats?clear&fs"

{"cluster_name":"elasticsearch","nodes":{"8f-ZbebNT2e23DArE6WnIw":{"timestamp":1391183667580,"name":"ELASTICSEARCH-4","transport_address":"inet[
192.168.248.172/192.168.248.172:9300
]","indices":{"docs":{"count":2180,"deleted":0},"store":{"size":"1.5mb","size_in_bytes":1631986,"throttle_time":"0s","throttle_time_in_millis":0},"indexing":{"index_total":0,"index_time":"0s","index_time_in_millis":0,"index_current":0,"delete_total":0,"delete_time":"0s","delete_time_in_millis":0,"delete_current":0},"get":{"total":0,"get_time":"0s","time_in_millis":0,"exists_total":0,"exists_time":"0s","exists_time_in_millis":0,"missing_total":0,"missing_time":"0s","missing_time_in_millis":0,"current":0},"search":{"open_contexts":0,"query_total":0,"query_time":"0s","query_time_in_millis":0,"query_current":0,"fetch_total":0,"fetch_time":"0s","fetch_time_in_millis":0,"fetch_current":0}}}}}

df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 15G 2.4G 12G 17% /
devtmpfs 743M 32K 743M 1% /dev
tmpfs 748M 0 748M 0% /dev/shm
tmpfs 748M 3.1M 744M 1% /run
tmpfs 748M 0 748M 0% /sys/fs/cgroup
tmpfs 748M 3.1M 744M 1% /var/run
tmpfs 748M 3.1M 744M 1% /var/lock
.host:/ 905G 467G 438G 52% /mnt/hgfs

On Friday, January 31, 2014 12:52:21 AM UTC-8, Boaz Leskes wrote:

Hi Tony,

I suspect this is an issue with the reporting of the Nodes Stats API
(based on sigar). Can you please post the output of calling:

curl -XGET "http://localhost:9200/_cluster/nodes/_local/stats?clear&fs"

&

df -h

on one of the machines 2-5?

Thx,
Boaz

On Thursday, January 30, 2014 11:39:48 PM UTC+1, Tony Su wrote:

Deployed in a VMware environment.
Data is being consumed and is successful from source to ES

source > logstash shipper(and parser) > redis > logstash > es cluster

ES Cluster
All openSUSE 13.1
Machine 1 - 4GB ram - 16GB HDD - openSUSE/LXDE -ES -Logstash - netcat
Machine 2 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
Machine 3 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
Machine 4 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
Machine 5 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES

Marvel reports the disk space likely correctly only on the initial
machine (8.8GB free)
For machines 2-5, free disk space is reported as 434.3 GB

For at least the next day, I'm going to be trying to interpret my
results based on the existing Marvel documentation, but this anomaly jumped
out at me.
I don't know how the disk free space is read by Marvel, maybe it's
using a method that doesn't understand a VMware virtual disk? Is pretty
ordinary, formatted ext4. Maybe a system level dependency is missing on
these machines?

Thx,
Tony

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c608547a-ca43-4433-9af5-cde2143203e4%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Boaz_Leskes · January 31, 2014, 5:38pm

Hi Tony,

I'm a bit confused. These numbers seem to add up? Both marvel, df -h and
the ES stats say you have ~430GB free on ".host:/", where ES is storing
it's data.

Boaz

On Fri, Jan 31, 2014 at 5:55 PM, Tony Su tonysu999@gmail.com wrote:

Sorry, you're right. Typo in my curl command.

corrected

{"cluster_name":"elasticsearch","nodes":{"8f-ZbebNT2e23DArE6WnIw":{"timestamp":1391187042684,"name":"ELASTICSEARCH-4","transport_address":"inet[
192.168.248.172/192.168.248.172:9300
]","fs":{"timestamp":1391187042685,"total":{"total":"904.3gb","total_in_bytes":971057917952,"free":"431.7gb","free_in_bytes":463607828480,"available":"431.7gb","available_in_bytes":463607828480},"data":[{"path":"/mnt/hgfs/DataShared/elasticsearch/nodes/0","mount":"/mnt/hgfs","dev":".host:/","total":"904.3gb","total_in_bytes":971057917952,"free":"431.7gb","free_in_bytes":463607828480,"available":"431.7gb","available_in_bytes":463607828480}]}}}}

On Friday, January 31, 2014 8:42:02 AM UTC-8, Boaz Leskes wrote:

Thx.

Something went wrong with the curl command as it returned indices stats
only (which is the default). The "&fs" should have caused it to return file
system stats. Perhaps something went wrong with the '&'. Can you double
check?

On Friday, January 31, 2014 5:10:02 PM UTC+1, Tony Su wrote:

SIGA is great if that is what you're using...
If you'd like I can also test installing that separately to see if it
makes a diff...
Tony

curl -XGET "http://localhost:9200/_cluster/nodes/_local/stats?clear&fs

"
{"cluster_name":"elasticsearch","nodes":{"8f-ZbebNT2e23DArE6WnIw":{"
timestamp":1391183667580,"name":"ELASTICSEARCH-4","
transport_address":"inet[192.168.248.172/192.168.248.172:9300
]","indices":{"docs":{"count":2180,"deleted":0},"
store":{"size":"1.5mb","size_in_bytes":1631986,"throttle_
time":"0s","throttle_time_in_millis":0},"indexing":{"index_
total":0,"index_time":"0s","index_time_in_millis":0,"
index_current":0,"delete_total":0,"delete_time":"0s","
delete_time_in_millis":0,"delete_current":0},"get":{"
total":0,"get_time":"0s","time_in_millis":0,"exists_
total":0,"exists_time":"0s","exists_time_in_millis":0,"
missing_total":0,"missing_time":"0s","missing_time_in_
millis":0,"current":0},"search":{"open_contexts":0,"
query_total":0,"query_time":"0s","query_time_in_millis":0,"
query_current":0,"fetch_total":0,"fetch_time":"0s","fetch_
time_in_millis":0,"fetch_current":0}}}}}

df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 15G 2.4G 12G 17% /
devtmpfs 743M 32K 743M 1% /dev
tmpfs 748M 0 748M 0% /dev/shm
tmpfs 748M 3.1M 744M 1% /run
tmpfs 748M 0 748M 0% /sys/fs/cgroup
tmpfs 748M 3.1M 744M 1% /var/run
tmpfs 748M 3.1M 744M 1% /var/lock
.host:/ 905G 467G 438G 52% /mnt/hgfs

On Friday, January 31, 2014 12:52:21 AM UTC-8, Boaz Leskes wrote:

Hi Tony,

I suspect this is an issue with the reporting of the Nodes Stats API
(based on sigar). Can you please post the output of calling:

curl -XGET "http://localhost:9200/_cluster/nodes/_local/stats?clear&fs"

&

df -h

on one of the machines 2-5?

Thx,
Boaz

On Thursday, January 30, 2014 11:39:48 PM UTC+1, Tony Su wrote:

Deployed in a VMware environment.
Data is being consumed and is successful from source to ES

source > logstash shipper(and parser) > redis > logstash > es cluster

ES Cluster
All openSUSE 13.1
Machine 1 - 4GB ram - 16GB HDD - openSUSE/LXDE -ES -Logstash - netcat
Machine 2 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
Machine 3 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
Machine 4 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
Machine 5 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES

Marvel reports the disk space likely correctly only on the initial
machine (8.8GB free)
For machines 2-5, free disk space is reported as 434.3 GB

For at least the next day, I'm going to be trying to interpret my
results based on the existing Marvel documentation, but this anomaly jumped
out at me.
I don't know how the disk free space is read by Marvel, maybe it's
using a method that doesn't understand a VMware virtual disk? Is pretty
ordinary, formatted ext4. Maybe a system level dependency is missing on
these machines?

Thx,
Tony

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/h5XbhxZCiIU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c608547a-ca43-4433-9af5-cde2143203e4%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKzwz0rCmRvPpJ_DxeM_AXBqWgikYF8L2nfnJS782JRE0K-N%3DQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Tony_Su · January 31, 2014, 6:52pm

Well, of course none of that is true...
you include the mounted share which includes the raw data which is
being input into the system.

So, I guess that's where the anomaly exists...
I doubt that the source where the raw data is coming from should be
considered part of the "working environment" of Elasticsearch... I assume
that "free space" should only include what ES should see as available to
run, create and store (which of course is exclusive of the raw data source).

But, that raises a curious observation...
On Machine 1 (where all data is imported into an ES node, also running the
LS shipper and indexers and Redis) space is probably being reported
correctly (approx. half of the local disk, and although of course is using
the same data mount share).

Considering the above, I guess technically it might be considered that
Machines 2-5 have access to the free disk space on the shared data mount
although I can't imagine how it would be used in a normal setting. Maybe
instead the ES paths should be parsed and only the partitions those paths
point to should be reported as contributing to available "free space?" Or,
maybe even more accurately only the ES Data Path partition should be
reported? Taking this idea maybe a step further, there might be a question
how to report the ES TMP path, if it's not tmpfs then it should also be
included?

Tony

On Friday, January 31, 2014 9:38:42 AM UTC-8, Boaz Leskes wrote:

Hi Tony,

I'm a bit confused. These numbers seem to add up? Both marvel, df -h and
the ES stats say you have ~430GB free on ".host:/", where ES is storing
it's data.

Boaz

On Fri, Jan 31, 2014 at 5:55 PM, Tony Su <tony...@gmail.com <javascript:>>wrote:

Sorry, you're right. Typo in my curl command.

corrected

{"cluster_name":"elasticsearch","nodes":{"8f-ZbebNT2e23DArE6WnIw":{"timestamp":1391187042684,"name":"ELASTICSEARCH-4","transport_address":"inet[
192.168.248.172/192.168.248.172:9300
]","fs":{"timestamp":1391187042685,"total":{"total":"904.3gb","total_in_bytes":971057917952,"free":"431.7gb","free_in_bytes":463607828480,"available":"431.7gb","available_in_bytes":463607828480},"data":[{"path":"/mnt/hgfs/DataShared/elasticsearch/nodes/0","mount":"/mnt/hgfs","dev":".host:/","total":"904.3gb","total_in_bytes":971057917952,"free":"431.7gb","free_in_bytes":463607828480,"available":"431.7gb","available_in_bytes":463607828480}]}}}}

On Friday, January 31, 2014 8:42:02 AM UTC-8, Boaz Leskes wrote:

Thx.

Something went wrong with the curl command as it returned indices stats
only (which is the default). The "&fs" should have caused it to return file
system stats. Perhaps something went wrong with the '&'. Can you double
check?

On Friday, January 31, 2014 5:10:02 PM UTC+1, Tony Su wrote:

SIGA is great if that is what you're using...
If you'd like I can also test installing that separately to see if it
makes a diff...
Tony

curl -XGET "http://localhost:9200/_cluster/nodes/_local/stats?

clear&fs"
{"cluster_name":"elasticsearch","nodes":{"8f-ZbebNT2e23DArE6WnIw":{"
timestamp":1391183667580,"name":"ELASTICSEARCH-4","
transport_address":"inet[192.168.248.172/192.168.248.172:9300
]","indices":{"docs":{"count":2180,"deleted":0},"
store":{"size":"1.5mb","size_in_bytes":1631986,"throttle_
time":"0s","throttle_time_in_millis":0},"indexing":{"index_
total":0,"index_time":"0s","index_time_in_millis":0,"
index_current":0,"delete_total":0,"delete_time":"0s","
delete_time_in_millis":0,"delete_current":0},"get":{"
total":0,"get_time":"0s","time_in_millis":0,"exists_
total":0,"exists_time":"0s","exists_time_in_millis":0,"
missing_total":0,"missing_time":"0s","missing_time_in_
millis":0,"current":0},"search":{"open_contexts":0,"
query_total":0,"query_time":"0s","query_time_in_millis":0,"
query_current":0,"fetch_total":0,"fetch_time":"0s","fetch_
time_in_millis":0,"fetch_current":0}}}}}

df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 15G 2.4G 12G 17% /
devtmpfs 743M 32K 743M 1% /dev
tmpfs 748M 0 748M 0% /dev/shm
tmpfs 748M 3.1M 744M 1% /run
tmpfs 748M 0 748M 0% /sys/fs/cgroup
tmpfs 748M 3.1M 744M 1% /var/run
tmpfs 748M 3.1M 744M 1% /var/lock
.host:/ 905G 467G 438G 52% /mnt/hgfs

On Friday, January 31, 2014 12:52:21 AM UTC-8, Boaz Leskes wrote:

Hi Tony,

I suspect this is an issue with the reporting of the Nodes Stats API
(based on sigar). Can you please post the output of calling:

curl -XGET "http://localhost:9200/_cluster/nodes/_local/stats?clear&fs
"

&

df -h

on one of the machines 2-5?

Thx,
Boaz

On Thursday, January 30, 2014 11:39:48 PM UTC+1, Tony Su wrote:

Deployed in a VMware environment.
Data is being consumed and is successful from source to ES

source > logstash shipper(and parser) > redis > logstash > es cluster

ES Cluster
All openSUSE 13.1
Machine 1 - 4GB ram - 16GB HDD - openSUSE/LXDE -ES -Logstash - netcat
Machine 2 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
Machine 3 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
Machine 4 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
Machine 5 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES

Marvel reports the disk space likely correctly only on the initial
machine (8.8GB free)
For machines 2-5, free disk space is reported as 434.3 GB

For at least the next day, I'm going to be trying to interpret my
results based on the existing Marvel documentation, but this anomaly jumped
out at me.
I don't know how the disk free space is read by Marvel, maybe it's
using a method that doesn't understand a VMware virtual disk? Is pretty
ordinary, formatted ext4. Maybe a system level dependency is missing on
these machines?

Thx,
Tony

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/h5XbhxZCiIU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c608547a-ca43-4433-9af5-cde2143203e4%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/134f1460-9605-4625-9779-b6a4355d81d7%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Boaz_Leskes · January 31, 2014, 7:49pm

Marvel measure the free space on the file system where the ES it monitors
keeps it's data. According to the node stats you sent, this
is /mnt/hgfs/DataShared/elasticsearch :

"data":[{"path":"/mnt/hgfs/DataShared/elasticsearch/nodes/0","mount":
"/mnt/hgfs","dev":".host:/","total":"904.3gb"

which is on a mount with a total size of 904GB.

Is this not correct?

On Friday, January 31, 2014 7:52:32 PM UTC+1, Tony Su wrote:

Well, of course none of that is true...
you include the mounted share which includes the raw data which
is being input into the system.

So, I guess that's where the anomaly exists...
I doubt that the source where the raw data is coming from should be
considered part of the "working environment" of Elasticsearch... I assume
that "free space" should only include what ES should see as available to
run, create and store (which of course is exclusive of the raw data source).

But, that raises a curious observation...
On Machine 1 (where all data is imported into an ES node, also running the
LS shipper and indexers and Redis) space is probably being reported
correctly (approx. half of the local disk, and although of course is using
the same data mount share).

Considering the above, I guess technically it might be considered that
Machines 2-5 have access to the free disk space on the shared data mount
although I can't imagine how it would be used in a normal setting. Maybe
instead the ES paths should be parsed and only the partitions those paths
point to should be reported as contributing to available "free space?" Or,
maybe even more accurately only the ES Data Path partition should be
reported? Taking this idea maybe a step further, there might be a question
how to report the ES TMP path, if it's not tmpfs then it should also be
included?

Tony

Tony

On Friday, January 31, 2014 9:38:42 AM UTC-8, Boaz Leskes wrote:

Hi Tony,

I'm a bit confused. These numbers seem to add up? Both marvel, df -h and
the ES stats say you have ~430GB free on ".host:/", where ES is storing
it's data.

Boaz

On Fri, Jan 31, 2014 at 5:55 PM, Tony Su tony...@gmail.com wrote:

Sorry, you're right. Typo in my curl command.

corrected

{"cluster_name":"elasticsearch","nodes":{"8f-ZbebNT2e23DArE6WnIw":{"timestamp":1391187042684,"name":"ELASTICSEARCH-4","transport_address":"inet[
192.168.248.172/192.168.248.172:9300
]","fs":{"timestamp":1391187042685,"total":{"total":"904.3gb","total_in_bytes":971057917952,"free":"431.7gb","free_in_bytes":463607828480,"available":"431.7gb","available_in_bytes":463607828480},"data":[{"path":"/mnt/hgfs/DataShared/elasticsearch/nodes/0","mount":"/mnt/hgfs","dev":".host:/","total":"904.3gb","total_in_bytes":971057917952,"free":"431.7gb","free_in_bytes":463607828480,"available":"431.7gb","available_in_bytes":463607828480}]}}}}

On Friday, January 31, 2014 8:42:02 AM UTC-8, Boaz Leskes wrote:

Thx.

Something went wrong with the curl command as it returned indices stats
only (which is the default). The "&fs" should have caused it to return file
system stats. Perhaps something went wrong with the '&'. Can you double
check?

On Friday, January 31, 2014 5:10:02 PM UTC+1, Tony Su wrote:

SIGA is great if that is what you're using...
If you'd like I can also test installing that separately to see if it
makes a diff...
Tony

curl -XGET "http://localhost:9200/_cluster/nodes/_local/stats?

clear&fs"
{"cluster_name":"elasticsearch","nodes":{"8f-ZbebNT2e23DArE6WnIw":{"
timestamp":1391183667580,"name":"ELASTICSEARCH-4","
transport_address":"inet[192.168.248.172/192.168.248.172:9300
]","indices":{"docs":{"count":2180,"deleted":0},"
store":{"size":"1.5mb","size_in_bytes":1631986,"throttle_
time":"0s","throttle_time_in_millis":0},"indexing":{"index_
total":0,"index_time":"0s","index_time_in_millis":0,"
index_current":0,"delete_total":0,"delete_time":"0s","
delete_time_in_millis":0,"delete_current":0},"get":{"
total":0,"get_time":"0s","time_in_millis":0,"exists_
total":0,"exists_time":"0s","exists_time_in_millis":0,"
missing_total":0,"missing_time":"0s","missing_time_in_
millis":0,"current":0},"search":{"open_contexts":0,"
query_total":0,"query_time":"0s","query_time_in_millis":0,"
query_current":0,"fetch_total":0,"fetch_time":"0s","fetch_
time_in_millis":0,"fetch_current":0}}}}}

df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 15G 2.4G 12G 17% /
devtmpfs 743M 32K 743M 1% /dev
tmpfs 748M 0 748M 0% /dev/shm
tmpfs 748M 3.1M 744M 1% /run
tmpfs 748M 0 748M 0% /sys/fs/cgroup
tmpfs 748M 3.1M 744M 1% /var/run
tmpfs 748M 3.1M 744M 1% /var/lock
.host:/ 905G 467G 438G 52% /mnt/hgfs

On Friday, January 31, 2014 12:52:21 AM UTC-8, Boaz Leskes wrote:

Hi Tony,

I suspect this is an issue with the reporting of the Nodes Stats API
(based on sigar). Can you please post the output of calling:

curl -XGET "http://localhost:9200/_cluster/nodes/_local/stats?
clear&fs"

&

df -h

on one of the machines 2-5?

Thx,
Boaz

On Thursday, January 30, 2014 11:39:48 PM UTC+1, Tony Su wrote:

Deployed in a VMware environment.
Data is being consumed and is successful from source to ES

source > logstash shipper(and parser) > redis > logstash > es cluster

ES Cluster
All openSUSE 13.1
Machine 1 - 4GB ram - 16GB HDD - openSUSE/LXDE -ES -Logstash - netcat
Machine 2 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
Machine 3 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
Machine 4 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
Machine 5 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES

Marvel reports the disk space likely correctly only on the initial
machine (8.8GB free)
For machines 2-5, free disk space is reported as 434.3 GB

For at least the next day, I'm going to be trying to interpret my
results based on the existing Marvel documentation, but this anomaly jumped
out at me.
I don't know how the disk free space is read by Marvel, maybe it's
using a method that doesn't understand a VMware virtual disk? Is pretty
ordinary, formatted ext4. Maybe a system level dependency is missing on
these machines?

Thx,
Tony

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/h5XbhxZCiIU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c608547a-ca43-4433-9af5-cde2143203e4%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1cacd9ed-66e6-4133-92fa-fd6f22059dd3%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Tony_Su · January 31, 2014, 10:54pm

You're absolutely right.
I forgot I had changed that setting long ago when I was still feeling my
way around what are in these directories.

Also resolved some other weirdness I was seeing which raises an off
the beat question...

Because all my Machine2-4 data directories were pointing to a shared
location, they all were writing to, and reading from the same.
Besides the disk space anomaly I noticed, I also noticed that the IOps for
these machines were also zero, I assume because whatever metadata they
required was already created and available through a mounted share and
didn't require a network transfer.

Without testing, superficially it looks like the nodes seemed to
perform without error.

So, can a "shared metadata repository" be a normal and perhaps advantageous
ES configuration or is separate metadata stores for each node still
recommended? If it works, I speculate that any I/O bottlenecks should be
avoided but could potentially be a point of failure (single repository).

Thx,
Tony

On Friday, January 31, 2014 11:49:10 AM UTC-8, Boaz Leskes wrote:

Marvel measure the free space on the file system where the ES it monitors
keeps it's data. According to the node stats you sent, this
is /mnt/hgfs/DataShared/elasticsearch :

"data":[{"path":"/mnt/hgfs/DataShared/elasticsearch/nodes/0","mount":
"/mnt/hgfs","dev":".host:/","total":"904.3gb"

which is on a mount with a total size of 904GB.

Is this not correct?

On Friday, January 31, 2014 7:52:32 PM UTC+1, Tony Su wrote:

Well, of course none of that is true...
you include the mounted share which includes the raw data which
is being input into the system.

So, I guess that's where the anomaly exists...
I doubt that the source where the raw data is coming from should be
considered part of the "working environment" of Elasticsearch... I assume
that "free space" should only include what ES should see as available to
run, create and store (which of course is exclusive of the raw data source).

But, that raises a curious observation...
On Machine 1 (where all data is imported into an ES node, also running
the LS shipper and indexers and Redis) space is probably being reported
correctly (approx. half of the local disk, and although of course is using
the same data mount share).

Considering the above, I guess technically it might be considered that
Machines 2-5 have access to the free disk space on the shared data mount
although I can't imagine how it would be used in a normal setting. Maybe
instead the ES paths should be parsed and only the partitions those paths
point to should be reported as contributing to available "free space?" Or,
maybe even more accurately only the ES Data Path partition should be
reported? Taking this idea maybe a step further, there might be a question
how to report the ES TMP path, if it's not tmpfs then it should also be
included?

Tony

Tony

On Friday, January 31, 2014 9:38:42 AM UTC-8, Boaz Leskes wrote:

Hi Tony,

I'm a bit confused. These numbers seem to add up? Both marvel, df -h and
the ES stats say you have ~430GB free on ".host:/", where ES is storing
it's data.

Boaz

On Fri, Jan 31, 2014 at 5:55 PM, Tony Su tony...@gmail.com wrote:

Sorry, you're right. Typo in my curl command.

corrected

{"cluster_name":"elasticsearch","nodes":{"8f-ZbebNT2e23DArE6WnIw":{"timestamp":1391187042684,"name":"ELASTICSEARCH-4","transport_address":"inet[
192.168.248.172/192.168.248.172:9300
]","fs":{"timestamp":1391187042685,"total":{"total":"904.3gb","total_in_bytes":971057917952,"free":"431.7gb","free_in_bytes":463607828480,"available":"431.7gb","available_in_bytes":463607828480},"data":[{"path":"/mnt/hgfs/DataShared/elasticsearch/nodes/0","mount":"/mnt/hgfs","dev":".host:/","total":"904.3gb","total_in_bytes":971057917952,"free":"431.7gb","free_in_bytes":463607828480,"available":"431.7gb","available_in_bytes":463607828480}]}}}}

On Friday, January 31, 2014 8:42:02 AM UTC-8, Boaz Leskes wrote:

Thx.

Something went wrong with the curl command as it returned indices
stats only (which is the default). The "&fs" should have caused it to
return file system stats. Perhaps something went wrong with the '&'. Can
you double check?

On Friday, January 31, 2014 5:10:02 PM UTC+1, Tony Su wrote:

SIGA is great if that is what you're using...
If you'd like I can also test installing that separately to see if it
makes a diff...
Tony

curl -XGET "http://localhost:9200/_cluster/nodes/_local/stats?

clear&fs"
{"cluster_name":"elasticsearch","nodes":{"8f-ZbebNT2e23DArE6WnIw":{"
timestamp":1391183667580,"name":"ELASTICSEARCH-4","
transport_address":"inet[192.168.248.172/192.168.248.172:9300
]","indices":{"docs":{"count":2180,"deleted":0},"
store":{"size":"1.5mb","size_in_bytes":1631986,"throttle_
time":"0s","throttle_time_in_millis":0},"indexing":{"index_
total":0,"index_time":"0s","index_time_in_millis":0,"
index_current":0,"delete_total":0,"delete_time":"0s","
delete_time_in_millis":0,"delete_current":0},"get":{"
total":0,"get_time":"0s","time_in_millis":0,"exists_
total":0,"exists_time":"0s","exists_time_in_millis":0,"
missing_total":0,"missing_time":"0s","missing_time_in_
millis":0,"current":0},"search":{"open_contexts":0,"
query_total":0,"query_time":"0s","query_time_in_millis":0,"
query_current":0,"fetch_total":0,"fetch_time":"0s","fetch_
time_in_millis":0,"fetch_current":0}}}}}

df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 15G 2.4G 12G 17% /
devtmpfs 743M 32K 743M 1% /dev
tmpfs 748M 0 748M 0% /dev/shm
tmpfs 748M 3.1M 744M 1% /run
tmpfs 748M 0 748M 0% /sys/fs/cgroup
tmpfs 748M 3.1M 744M 1% /var/run
tmpfs 748M 3.1M 744M 1% /var/lock
.host:/ 905G 467G 438G 52% /mnt/hgfs

On Friday, January 31, 2014 12:52:21 AM UTC-8, Boaz Leskes wrote:

Hi Tony,

I suspect this is an issue with the reporting of the Nodes Stats API
(based on sigar). Can you please post the output of calling:

curl -XGET "http://localhost:9200/_cluster/nodes/_local/stats?
clear&fs"

&

df -h

on one of the machines 2-5?

Thx,
Boaz

On Thursday, January 30, 2014 11:39:48 PM UTC+1, Tony Su wrote:

Deployed in a VMware environment.
Data is being consumed and is successful from source to ES

source > logstash shipper(and parser) > redis > logstash > es
cluster

ES Cluster
All openSUSE 13.1
Machine 1 - 4GB ram - 16GB HDD - openSUSE/LXDE -ES -Logstash -
netcat
Machine 2 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
Machine 3 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
Machine 4 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
Machine 5 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES

Marvel reports the disk space likely correctly only on the initial
machine (8.8GB free)
For machines 2-5, free disk space is reported as 434.3 GB

For at least the next day, I'm going to be trying to interpret my
results based on the existing Marvel documentation, but this anomaly jumped
out at me.
I don't know how the disk free space is read by Marvel, maybe it's
using a method that doesn't understand a VMware virtual disk? Is pretty
ordinary, formatted ext4. Maybe a system level dependency is missing on
these machines?

Thx,
Tony

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/h5XbhxZCiIU/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c608547a-ca43-4433-9af5-cde2143203e4%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/594a220c-a9c8-4359-b40f-759171acbf02%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.