# Marvel reporting incorrect free disk space?
**URL:** https://discuss.elastic.co/t/marvel-reporting-incorrect-free-disk-space/15515
**Category:** Elasticsearch
**Created:** [January 30, 2014, 10:39pm UTC](https://discuss.elastic.co/t/marvel-reporting-incorrect-free-disk-space/15515 "2014-01-30T22:39:48Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@Tony\_Su](https://discuss.elastic.co/u/Tony_Su)
#### Post date: [January 30, 2014, 10:39pm UTC](https://discuss.elastic.co/t/marvel-reporting-incorrect-free-disk-space/15515/1 "2014-01-30T22:39:48Z")
Deployed in a VMware environment.
Data is being consumed and is successful from source to ES
source \> logstash shipper(and parser) \> redis \> logstash \> es cluster
ES Cluster
All openSUSE 13.1
Machine 1 - 4GB ram - 16GB HDD - openSUSE/LXDE -ES -Logstash - netcat
Machine 2 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
Machine 3 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
Machine 4 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
Machine 5 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
Marvel reports the disk space likely correctly only on the initial machine
(8.8GB free)
For machines 2-5, free disk space is reported as 434.3 GB
For at least the next day, I'm going to be trying to interpret my results
based on the existing Marvel documentation, but this anomaly jumped out at
me.
I don't know how the disk free space is read by Marvel, maybe it's using a
method that doesn't understand a VMware virtual disk? Is pretty ordinary,
formatted ext4. Maybe a system level dependency is missing on these
machines?
Thx,
Tony
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1904b915-a5a6-4712-b6f7-9635755d1ff9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1904b915-a5a6-4712-b6f7-9635755d1ff9%40googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Boaz\_Leskes](https://discuss.elastic.co/u/Boaz_Leskes)
#### Post date: [January 31, 2014, 8:52am UTC](https://discuss.elastic.co/t/marvel-reporting-incorrect-free-disk-space/15515/2 "2014-01-31T08:52:21Z")
Hi Tony,
I suspect this is an issue with the reporting of the Nodes Stats API (based
on sigar). Can you please post the output of calling:
curl -XGET "[http://localhost:9200/\_cluster/nodes/\_local/stats?clear&fs](http://localhost:9200/_cluster/nodes/_local/stats?clear&fs)"
&
df -h
on one of the machines 2-5?
Thx,
Boaz
On Thursday, January 30, 2014 11:39:48 PM UTC+1, Tony Su wrote:
> Deployed in a VMware environment.
> Data is being consumed and is successful from source to ES
>
> source \> logstash shipper(and parser) \> redis \> logstash \> es cluster
>
> ES Cluster
> All openSUSE 13.1
> Machine 1 - 4GB ram - 16GB HDD - openSUSE/LXDE -ES -Logstash - netcat
> Machine 2 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
> Machine 3 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
> Machine 4 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
> Machine 5 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
>
> Marvel reports the disk space likely correctly only on the initial machine
> (8.8GB free)
> For machines 2-5, free disk space is reported as 434.3 GB
>
> For at least the next day, I'm going to be trying to interpret my results
> based on the existing Marvel documentation, but this anomaly jumped out at
> me.
> I don't know how the disk free space is read by Marvel, maybe it's using a
> method that doesn't understand a VMware virtual disk? Is pretty ordinary,
> formatted ext4. Maybe a system level dependency is missing on these
> machines?
>
> Thx,
> Tony
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/28c0440b-54eb-4591-9c15-fe2b5f253632%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/28c0440b-54eb-4591-9c15-fe2b5f253632%40googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Tony\_Su](https://discuss.elastic.co/u/Tony_Su)
#### Post date: [January 31, 2014, 4:10pm UTC](https://discuss.elastic.co/t/marvel-reporting-incorrect-free-disk-space/15515/3 "2014-01-31T16:10:02Z")
SIGA is great if that is what you're using...
If you'd like I can also test installing that separately to see if it makes
a diff...
Tony
# curl -XGET "[http://localhost:9200/\_cluster/nodes/\_local/stats?clear&fs](http://localhost:9200/_cluster/nodes/_local/stats?clear&fs)"
{"cluster\_name":"elasticsearch","nodes":{"8f-ZbebNT2e23DArE6WnIw":{"timestamp":1391183667580,"name":"ELASTICSEARCH-4","transport\_address":"inet[192.168.248.172/192.168.248.172:9300]","indices":{"docs":{"count":2180,"deleted":0},"store":{"size":"1.5mb","size\_in\_bytes":1631986,"throttle\_time":"0s","throttle\_time\_in\_millis":0},"indexing":{"index\_total":0,"index\_time":"0s","index\_time\_in\_millis":0,"index\_current":0,"delete\_total":0,"delete\_time":"0s","delete\_time\_in\_millis":0,"delete\_current":0},"get":{"total":0,"get\_time":"0s","time\_in\_millis":0,"exists\_total":0,"exists\_time":"0s","exists\_time\_in\_millis":0,"missing\_total":0,"missing\_time":"0s","missing\_time\_in\_millis":0,"current":0},"search":{"open\_contexts":0,"query\_total":0,"query\_time":"0s","query\_time\_in\_millis":0,"query\_current":0,"fetch\_total":0,"fetch\_time":"0s","fetch\_time\_in\_millis":0,"fetch\_current":0}}}}}
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 15G 2.4G 12G 17% /
devtmpfs 743M 32K 743M 1% /dev
tmpfs 748M 0 748M 0% /dev/shm
tmpfs 748M 3.1M 744M 1% /run
tmpfs 748M 0 748M 0% /sys/fs/cgroup
tmpfs 748M 3.1M 744M 1% /var/run
tmpfs 748M 3.1M 744M 1% /var/lock
.host:/ 905G 467G 438G 52% /mnt/hgfs
On Friday, January 31, 2014 12:52:21 AM UTC-8, Boaz Leskes wrote:
> Hi Tony,
>
> I suspect this is an issue with the reporting of the Nodes Stats API
> (based on sigar). Can you please post the output of calling:
>
> curl -XGET "[http://localhost:9200/\_cluster/nodes/\_local/stats?clear&fs](http://localhost:9200/_cluster/nodes/_local/stats?clear&fs)"
>
> &
>
> df -h
>
> on one of the machines 2-5?
>
> Thx,
> Boaz
>
> On Thursday, January 30, 2014 11:39:48 PM UTC+1, Tony Su wrote:
>
> > Deployed in a VMware environment.
> > Data is being consumed and is successful from source to ES
> >
> > source \> logstash shipper(and parser) \> redis \> logstash \> es cluster
> >
> > ES Cluster
> > All openSUSE 13.1
> > Machine 1 - 4GB ram - 16GB HDD - openSUSE/LXDE -ES -Logstash - netcat
> > Machine 2 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
> > Machine 3 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
> > Machine 4 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
> > Machine 5 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
> >
> > Marvel reports the disk space likely correctly only on the initial
> > machine (8.8GB free)
> > For machines 2-5, free disk space is reported as 434.3 GB
> >
> > For at least the next day, I'm going to be trying to interpret my results
> > based on the existing Marvel documentation, but this anomaly jumped out at
> > me.
> > I don't know how the disk free space is read by Marvel, maybe it's using
> > a method that doesn't understand a VMware virtual disk? Is pretty ordinary,
> > formatted ext4. Maybe a system level dependency is missing on these
> > machines?
> >
> > Thx,
> > Tony
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5d930324-3d15-4a30-9f7c-c70100d201de%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5d930324-3d15-4a30-9f7c-c70100d201de%40googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Boaz\_Leskes](https://discuss.elastic.co/u/Boaz_Leskes)
#### Post date: [January 31, 2014, 4:42pm UTC](https://discuss.elastic.co/t/marvel-reporting-incorrect-free-disk-space/15515/4 "2014-01-31T16:42:02Z")
Thx.
Something went wrong with the curl command as it returned indices stats
only (which is the default). The "&fs" should have caused it to return file
system stats. Perhaps something went wrong with the '&'. Can you double
check?
On Friday, January 31, 2014 5:10:02 PM UTC+1, Tony Su wrote:
> SIGA is great if that is what you're using...
> If you'd like I can also test installing that separately to see if it
> makes a diff...
> Tony
>
> # curl -XGET "[http://localhost:9200/\_cluster/nodes/\_local/stats?clear&fs](http://localhost:9200/_cluster/nodes/_local/stats?clear&fs)"
>
> {"cluster\_name":"elasticsearch","nodes":{"8f-ZbebNT2e23DArE6WnIw":{"timestamp":1391183667580,"name":"ELASTICSEARCH-4","transport\_address":"inet[
> 192.168.248.172/192.168.248.172:9300
> ]","indices":{"docs":{"count":2180,"deleted":0},"store":{"size":"1.5mb","size\_in\_bytes":1631986,"throttle\_time":"0s","throttle\_time\_in\_millis":0},"indexing":{"index\_total":0,"index\_time":"0s","index\_time\_in\_millis":0,"index\_current":0,"delete\_total":0,"delete\_time":"0s","delete\_time\_in\_millis":0,"delete\_current":0},"get":{"total":0,"get\_time":"0s","time\_in\_millis":0,"exists\_total":0,"exists\_time":"0s","exists\_time\_in\_millis":0,"missing\_total":0,"missing\_time":"0s","missing\_time\_in\_millis":0,"current":0},"search":{"open\_contexts":0,"query\_total":0,"query\_time":"0s","query\_time\_in\_millis":0,"query\_current":0,"fetch\_total":0,"fetch\_time":"0s","fetch\_time\_in\_millis":0,"fetch\_current":0}}}}}
>
> df -h
> Filesystem Size Used Avail Use% Mounted on
> /dev/sda2 15G 2.4G 12G 17% /
> devtmpfs 743M 32K 743M 1% /dev
> tmpfs 748M 0 748M 0% /dev/shm
> tmpfs 748M 3.1M 744M 1% /run
> tmpfs 748M 0 748M 0% /sys/fs/cgroup
> tmpfs 748M 3.1M 744M 1% /var/run
> tmpfs 748M 3.1M 744M 1% /var/lock
> .host:/ 905G 467G 438G 52% /mnt/hgfs
>
> On Friday, January 31, 2014 12:52:21 AM UTC-8, Boaz Leskes wrote:
>
> > Hi Tony,
> >
> > I suspect this is an issue with the reporting of the Nodes Stats API
> > (based on sigar). Can you please post the output of calling:
> >
> > curl -XGET "[http://localhost:9200/\_cluster/nodes/\_local/stats?clear&fs](http://localhost:9200/_cluster/nodes/_local/stats?clear&fs)"
> >
> > &
> >
> > df -h
> >
> > on one of the machines 2-5?
> >
> > Thx,
> > Boaz
> >
> > On Thursday, January 30, 2014 11:39:48 PM UTC+1, Tony Su wrote:
> >
> > > Deployed in a VMware environment.
> > > Data is being consumed and is successful from source to ES
> > >
> > > source \> logstash shipper(and parser) \> redis \> logstash \> es cluster
> > >
> > > ES Cluster
> > > All openSUSE 13.1
> > > Machine 1 - 4GB ram - 16GB HDD - openSUSE/LXDE -ES -Logstash - netcat
> > > Machine 2 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
> > > Machine 3 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
> > > Machine 4 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
> > > Machine 5 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
> > >
> > > Marvel reports the disk space likely correctly only on the initial
> > > machine (8.8GB free)
> > > For machines 2-5, free disk space is reported as 434.3 GB
> > >
> > > For at least the next day, I'm going to be trying to interpret my
> > > results based on the existing Marvel documentation, but this anomaly jumped
> > > out at me.
> > > I don't know how the disk free space is read by Marvel, maybe it's using
> > > a method that doesn't understand a VMware virtual disk? Is pretty ordinary,
> > > formatted ext4. Maybe a system level dependency is missing on these
> > > machines?
> > >
> > > Thx,
> > > Tony
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/fcd17f7e-13b8-4f88-ad63-76dde148975f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/fcd17f7e-13b8-4f88-ad63-76dde148975f%40googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Tony\_Su](https://discuss.elastic.co/u/Tony_Su)
#### Post date: [January 31, 2014, 4:55pm UTC](https://discuss.elastic.co/t/marvel-reporting-incorrect-free-disk-space/15515/5 "2014-01-31T16:55:00Z")
Sorry, you're right. Typo in my curl command.
corrected
{"cluster\_name":"elasticsearch","nodes":{"8f-ZbebNT2e23DArE6WnIw":{"timestamp":1391187042684,"name":"ELASTICSEARCH-4","transport\_address":"inet[192.168.248.172/192.168.248.172:9300]","fs":{"timestamp":1391187042685,"total":{"total":"904.3gb","total\_in\_bytes":971057917952,"free":"431.7gb","free\_in\_bytes":463607828480,"available":"431.7gb","available\_in\_bytes":463607828480},"data":[{"path":"/mnt/hgfs/DataShared/elasticsearch/nodes/0","mount":"/mnt/hgfs","dev":".host:/","total":"904.3gb","total\_in\_bytes":971057917952,"free":"431.7gb","free\_in\_bytes":463607828480,"available":"431.7gb","available\_in\_bytes":463607828480}]}}}}
On Friday, January 31, 2014 8:42:02 AM UTC-8, Boaz Leskes wrote:
> Thx.
>
> Something went wrong with the curl command as it returned indices stats
> only (which is the default). The "&fs" should have caused it to return file
> system stats. Perhaps something went wrong with the '&'. Can you double
> check?
>
> On Friday, January 31, 2014 5:10:02 PM UTC+1, Tony Su wrote:
>
> > SIGA is great if that is what you're using...
> > If you'd like I can also test installing that separately to see if it
> > makes a diff...
> > Tony
> >
> > # curl -XGET "[http://localhost:9200/\_cluster/nodes/\_local/stats?clear&fs](http://localhost:9200/_cluster/nodes/_local/stats?clear&fs)"
> >
> > {"cluster\_name":"elasticsearch","nodes":{"8f-ZbebNT2e23DArE6WnIw":{"timestamp":1391183667580,"name":"ELASTICSEARCH-4","transport\_address":"inet[
> > 192.168.248.172/192.168.248.172:9300
> > ]","indices":{"docs":{"count":2180,"deleted":0},"store":{"size":"1.5mb","size\_in\_bytes":1631986,"throttle\_time":"0s","throttle\_time\_in\_millis":0},"indexing":{"index\_total":0,"index\_time":"0s","index\_time\_in\_millis":0,"index\_current":0,"delete\_total":0,"delete\_time":"0s","delete\_time\_in\_millis":0,"delete\_current":0},"get":{"total":0,"get\_time":"0s","time\_in\_millis":0,"exists\_total":0,"exists\_time":"0s","exists\_time\_in\_millis":0,"missing\_total":0,"missing\_time":"0s","missing\_time\_in\_millis":0,"current":0},"search":{"open\_contexts":0,"query\_total":0,"query\_time":"0s","query\_time\_in\_millis":0,"query\_current":0,"fetch\_total":0,"fetch\_time":"0s","fetch\_time\_in\_millis":0,"fetch\_current":0}}}}}
> >
> > df -h
> > Filesystem Size Used Avail Use% Mounted on
> > /dev/sda2 15G 2.4G 12G 17% /
> > devtmpfs 743M 32K 743M 1% /dev
> > tmpfs 748M 0 748M 0% /dev/shm
> > tmpfs 748M 3.1M 744M 1% /run
> > tmpfs 748M 0 748M 0% /sys/fs/cgroup
> > tmpfs 748M 3.1M 744M 1% /var/run
> > tmpfs 748M 3.1M 744M 1% /var/lock
> > .host:/ 905G 467G 438G 52% /mnt/hgfs
> >
> > On Friday, January 31, 2014 12:52:21 AM UTC-8, Boaz Leskes wrote:
> >
> > > Hi Tony,
> > >
> > > I suspect this is an issue with the reporting of the Nodes Stats API
> > > (based on sigar). Can you please post the output of calling:
> > >
> > > curl -XGET "[http://localhost:9200/\_cluster/nodes/\_local/stats?clear&fs](http://localhost:9200/_cluster/nodes/_local/stats?clear&fs)"
> > >
> > > &
> > >
> > > df -h
> > >
> > > on one of the machines 2-5?
> > >
> > > Thx,
> > > Boaz
> > >
> > > On Thursday, January 30, 2014 11:39:48 PM UTC+1, Tony Su wrote:
> > >
> > > > Deployed in a VMware environment.
> > > > Data is being consumed and is successful from source to ES
> > > >
> > > > source \> logstash shipper(and parser) \> redis \> logstash \> es cluster
> > > >
> > > > ES Cluster
> > > > All openSUSE 13.1
> > > > Machine 1 - 4GB ram - 16GB HDD - openSUSE/LXDE -ES -Logstash - netcat
> > > > Machine 2 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
> > > > Machine 3 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
> > > > Machine 4 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
> > > > Machine 5 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
> > > >
> > > > Marvel reports the disk space likely correctly only on the initial
> > > > machine (8.8GB free)
> > > > For machines 2-5, free disk space is reported as 434.3 GB
> > > >
> > > > For at least the next day, I'm going to be trying to interpret my
> > > > results based on the existing Marvel documentation, but this anomaly jumped
> > > > out at me.
> > > > I don't know how the disk free space is read by Marvel, maybe it's
> > > > using a method that doesn't understand a VMware virtual disk? Is pretty
> > > > ordinary, formatted ext4. Maybe a system level dependency is missing on
> > > > these machines?
> > > >
> > > > Thx,
> > > > Tony
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/c608547a-ca43-4433-9af5-cde2143203e4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c608547a-ca43-4433-9af5-cde2143203e4%40googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Boaz\_Leskes](https://discuss.elastic.co/u/Boaz_Leskes)
#### Post date: [January 31, 2014, 5:38pm UTC](https://discuss.elastic.co/t/marvel-reporting-incorrect-free-disk-space/15515/6 "2014-01-31T17:38:42Z")
Hi Tony,
I'm a bit confused. These numbers seem to add up? Both marvel, df -h and
the ES stats say you have ~430GB free on ".host:/", where ES is storing
it's data.
Boaz
On Fri, Jan 31, 2014 at 5:55 PM, Tony Su [tonysu999@gmail.com](mailto:tonysu999@gmail.com) wrote:
> Sorry, you're right. Typo in my curl command.
>
> corrected
>
> {"cluster\_name":"elasticsearch","nodes":{"8f-ZbebNT2e23DArE6WnIw":{"timestamp":1391187042684,"name":"ELASTICSEARCH-4","transport\_address":"inet[
> 192.168.248.172/192.168.248.172:9300
> ]","fs":{"timestamp":1391187042685,"total":{"total":"904.3gb","total\_in\_bytes":971057917952,"free":"431.7gb","free\_in\_bytes":463607828480,"available":"431.7gb","available\_in\_bytes":463607828480},"data":[{"path":"/mnt/hgfs/DataShared/elasticsearch/nodes/0","mount":"/mnt/hgfs","dev":".host:/","total":"904.3gb","total\_in\_bytes":971057917952,"free":"431.7gb","free\_in\_bytes":463607828480,"available":"431.7gb","available\_in\_bytes":463607828480}]}}}}
>
> On Friday, January 31, 2014 8:42:02 AM UTC-8, Boaz Leskes wrote:
>
> > Thx.
> >
> > Something went wrong with the curl command as it returned indices stats
> > only (which is the default). The "&fs" should have caused it to return file
> > system stats. Perhaps something went wrong with the '&'. Can you double
> > check?
> >
> > On Friday, January 31, 2014 5:10:02 PM UTC+1, Tony Su wrote:
> >
> > > SIGA is great if that is what you're using...
> > > If you'd like I can also test installing that separately to see if it
> > > makes a diff...
> > > Tony
> > >
> > > # curl -XGET "[http://localhost:9200/\_cluster/nodes/\_local/stats?clear&fs](http://localhost:9200/_cluster/nodes/_local/stats?clear&fs)
> > >
> > > "
> > > {"cluster\_name":"elasticsearch","nodes":{"8f-ZbebNT2e23DArE6WnIw":{"
> > > timestamp":1391183667580,"name":"ELASTICSEARCH-4","
> > > transport\_address":"inet[192.168.248.172/192.168.248.172:9300
> > > ]","indices":{"docs":{"count":2180,"deleted":0},"
> > > store":{"size":"1.5mb","size\_in\_bytes":1631986,"throttle\_
> > > time":"0s","throttle\_time\_in\_millis":0},"indexing":{"index\_
> > > total":0,"index\_time":"0s","index\_time\_in\_millis":0,"
> > > index\_current":0,"delete\_total":0,"delete\_time":"0s","
> > > delete\_time\_in\_millis":0,"delete\_current":0},"get":{"
> > > total":0,"get\_time":"0s","time\_in\_millis":0,"exists\_
> > > total":0,"exists\_time":"0s","exists\_time\_in\_millis":0,"
> > > missing\_total":0,"missing\_time":"0s","missing\_time\_in\_
> > > millis":0,"current":0},"search":{"open\_contexts":0,"
> > > query\_total":0,"query\_time":"0s","query\_time\_in\_millis":0,"
> > > query\_current":0,"fetch\_total":0,"fetch\_time":"0s","fetch\_
> > > time\_in\_millis":0,"fetch\_current":0}}}}}
> > >
> > > df -h
> > > Filesystem Size Used Avail Use% Mounted on
> > > /dev/sda2 15G 2.4G 12G 17% /
> > > devtmpfs 743M 32K 743M 1% /dev
> > > tmpfs 748M 0 748M 0% /dev/shm
> > > tmpfs 748M 3.1M 744M 1% /run
> > > tmpfs 748M 0 748M 0% /sys/fs/cgroup
> > > tmpfs 748M 3.1M 744M 1% /var/run
> > > tmpfs 748M 3.1M 744M 1% /var/lock
> > > .host:/ 905G 467G 438G 52% /mnt/hgfs
> > >
> > > On Friday, January 31, 2014 12:52:21 AM UTC-8, Boaz Leskes wrote:
> > >
> > > > Hi Tony,
> > > >
> > > > I suspect this is an issue with the reporting of the Nodes Stats API
> > > > (based on sigar). Can you please post the output of calling:
> > > >
> > > > curl -XGET "[http://localhost:9200/\_cluster/nodes/\_local/stats?clear&fs](http://localhost:9200/_cluster/nodes/_local/stats?clear&fs)"
> > > >
> > > > &
> > > >
> > > > df -h
> > > >
> > > > on one of the machines 2-5?
> > > >
> > > > Thx,
> > > > Boaz
> > > >
> > > > On Thursday, January 30, 2014 11:39:48 PM UTC+1, Tony Su wrote:
> > > >
> > > > > Deployed in a VMware environment.
> > > > > Data is being consumed and is successful from source to ES
> > > > >
> > > > > source \> logstash shipper(and parser) \> redis \> logstash \> es cluster
> > > > >
> > > > > ES Cluster
> > > > > All openSUSE 13.1
> > > > > Machine 1 - 4GB ram - 16GB HDD - openSUSE/LXDE -ES -Logstash - netcat
> > > > > Machine 2 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
> > > > > Machine 3 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
> > > > > Machine 4 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
> > > > > Machine 5 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
> > > > >
> > > > > Marvel reports the disk space likely correctly only on the initial
> > > > > machine (8.8GB free)
> > > > > For machines 2-5, free disk space is reported as 434.3 GB
> > > > >
> > > > > For at least the next day, I'm going to be trying to interpret my
> > > > > results based on the existing Marvel documentation, but this anomaly jumped
> > > > > out at me.
> > > > > I don't know how the disk free space is read by Marvel, maybe it's
> > > > > using a method that doesn't understand a VMware virtual disk? Is pretty
> > > > > ordinary, formatted ext4. Maybe a system level dependency is missing on
> > > > > these machines?
> > > > >
> > > > > Thx,
> > > > > Tony
> > > >
> > > > --
> > > > You received this message because you are subscribed to a topic in the
> > > > Google Groups "elasticsearch" group.
> > > > To unsubscribe from this topic, visit
> > > > [https://groups.google.com/d/topic/elasticsearch/h5XbhxZCiIU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/h5XbhxZCiIU/unsubscribe).
> > > > To unsubscribe from this group and all its topics, send an email to
> > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > > To view this discussion on the web visit
> > > > [https://groups.google.com/d/msgid/elasticsearch/c608547a-ca43-4433-9af5-cde2143203e4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c608547a-ca43-4433-9af5-cde2143203e4%40googlegroups.com)
> > > > .
> > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKzwz0rCmRvPpJ\_DxeM\_AXBqWgikYF8L2nfnJS782JRE0K-N%3DQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKzwz0rCmRvPpJ_DxeM_AXBqWgikYF8L2nfnJS782JRE0K-N%3DQ%40mail.gmail.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Tony\_Su](https://discuss.elastic.co/u/Tony_Su)
#### Post date: [January 31, 2014, 6:52pm UTC](https://discuss.elastic.co/t/marvel-reporting-incorrect-free-disk-space/15515/7 "2014-01-31T18:52:32Z")
Well, of course none of that is true...
you include the mounted share which includes the raw data which is
being input into the system.
So, I guess that's where the anomaly exists...
I doubt that the source where the raw data is coming from should be
considered part of the "working environment" of Elasticsearch... I assume
that "free space" should only include what ES should see as available to
run, create and store (which of course is exclusive of the raw data source).
But, that raises a curious observation...
On Machine 1 (where all data is imported into an ES node, also running the
LS shipper and indexers and Redis) space is probably being reported
correctly (approx. half of the local disk, and although of course is using
the same data mount share).
Considering the above, I guess technically it might be considered that
Machines 2-5 have access to the free disk space on the shared data mount
although I can't imagine how it would be used in a normal setting. Maybe
instead the ES paths should be parsed and only the partitions those paths
point to should be reported as contributing to available "free space?" Or,
maybe even more accurately only the ES Data Path partition should be
reported? Taking this idea maybe a step further, there might be a question
how to report the ES TMP path, if it's not tmpfs then it should also be
included?
Tony
Tony
On Friday, January 31, 2014 9:38:42 AM UTC-8, Boaz Leskes wrote:
> Hi Tony,
>
> I'm a bit confused. These numbers seem to add up? Both marvel, df -h and
> the ES stats say you have ~430GB free on ".host:/", where ES is storing
> it's data.
>
> Boaz
>
> On Fri, Jan 31, 2014 at 5:55 PM, Tony Su \<[tony...@gmail.com](mailto:tony...@gmail.com) \\>wrote:
>
> > Sorry, you're right. Typo in my curl command.
> >
> > corrected
> >
> > {"cluster\_name":"elasticsearch","nodes":{"8f-ZbebNT2e23DArE6WnIw":{"timestamp":1391187042684,"name":"ELASTICSEARCH-4","transport\_address":"inet[
> > 192.168.248.172/192.168.248.172:9300
> > ]","fs":{"timestamp":1391187042685,"total":{"total":"904.3gb","total\_in\_bytes":971057917952,"free":"431.7gb","free\_in\_bytes":463607828480,"available":"431.7gb","available\_in\_bytes":463607828480},"data":[{"path":"/mnt/hgfs/DataShared/elasticsearch/nodes/0","mount":"/mnt/hgfs","dev":".host:/","total":"904.3gb","total\_in\_bytes":971057917952,"free":"431.7gb","free\_in\_bytes":463607828480,"available":"431.7gb","available\_in\_bytes":463607828480}]}}}}
> >
> > On Friday, January 31, 2014 8:42:02 AM UTC-8, Boaz Leskes wrote:
> >
> > > Thx.
> > >
> > > Something went wrong with the curl command as it returned indices stats
> > > only (which is the default). The "&fs" should have caused it to return file
> > > system stats. Perhaps something went wrong with the '&'. Can you double
> > > check?
> > >
> > > On Friday, January 31, 2014 5:10:02 PM UTC+1, Tony Su wrote:
> > >
> > > > SIGA is great if that is what you're using...
> > > > If you'd like I can also test installing that separately to see if it
> > > > makes a diff...
> > > > Tony
> > > >
> > > > # curl -XGET "[http://localhost:9200/\_cluster/nodes/\_local/stats](http://localhost:9200/_cluster/nodes/_local/stats)?
> > > >
> > > > clear&fs"
> > > > {"cluster\_name":"elasticsearch","nodes":{"8f-ZbebNT2e23DArE6WnIw":{"
> > > > timestamp":1391183667580,"name":"ELASTICSEARCH-4","
> > > > transport\_address":"inet[192.168.248.172/192.168.248.172:9300
> > > > ]","indices":{"docs":{"count":2180,"deleted":0},"
> > > > store":{"size":"1.5mb","size\_in\_bytes":1631986,"throttle\_
> > > > time":"0s","throttle\_time\_in\_millis":0},"indexing":{"index\_
> > > > total":0,"index\_time":"0s","index\_time\_in\_millis":0,"
> > > > index\_current":0,"delete\_total":0,"delete\_time":"0s","
> > > > delete\_time\_in\_millis":0,"delete\_current":0},"get":{"
> > > > total":0,"get\_time":"0s","time\_in\_millis":0,"exists\_
> > > > total":0,"exists\_time":"0s","exists\_time\_in\_millis":0,"
> > > > missing\_total":0,"missing\_time":"0s","missing\_time\_in\_
> > > > millis":0,"current":0},"search":{"open\_contexts":0,"
> > > > query\_total":0,"query\_time":"0s","query\_time\_in\_millis":0,"
> > > > query\_current":0,"fetch\_total":0,"fetch\_time":"0s","fetch\_
> > > > time\_in\_millis":0,"fetch\_current":0}}}}}
> > > >
> > > > df -h
> > > > Filesystem Size Used Avail Use% Mounted on
> > > > /dev/sda2 15G 2.4G 12G 17% /
> > > > devtmpfs 743M 32K 743M 1% /dev
> > > > tmpfs 748M 0 748M 0% /dev/shm
> > > > tmpfs 748M 3.1M 744M 1% /run
> > > > tmpfs 748M 0 748M 0% /sys/fs/cgroup
> > > > tmpfs 748M 3.1M 744M 1% /var/run
> > > > tmpfs 748M 3.1M 744M 1% /var/lock
> > > > .host:/ 905G 467G 438G 52% /mnt/hgfs
> > > >
> > > > On Friday, January 31, 2014 12:52:21 AM UTC-8, Boaz Leskes wrote:
> > > >
> > > > > Hi Tony,
> > > > >
> > > > > I suspect this is an issue with the reporting of the Nodes Stats API
> > > > > (based on sigar). Can you please post the output of calling:
> > > > >
> > > > > curl -XGET "[http://localhost:9200/\_cluster/nodes/\_local/stats?clear&fs](http://localhost:9200/_cluster/nodes/_local/stats?clear&fs)
> > > > > "
> > > > >
> > > > > &
> > > > >
> > > > > df -h
> > > > >
> > > > > on one of the machines 2-5?
> > > > >
> > > > > Thx,
> > > > > Boaz
> > > > >
> > > > > On Thursday, January 30, 2014 11:39:48 PM UTC+1, Tony Su wrote:
> > > > >
> > > > > > Deployed in a VMware environment.
> > > > > > Data is being consumed and is successful from source to ES
> > > > > >
> > > > > > source \> logstash shipper(and parser) \> redis \> logstash \> es cluster
> > > > > >
> > > > > > ES Cluster
> > > > > > All openSUSE 13.1
> > > > > > Machine 1 - 4GB ram - 16GB HDD - openSUSE/LXDE -ES -Logstash - netcat
> > > > > > Machine 2 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
> > > > > > Machine 3 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
> > > > > > Machine 4 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
> > > > > > Machine 5 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
> > > > > >
> > > > > > Marvel reports the disk space likely correctly only on the initial
> > > > > > machine (8.8GB free)
> > > > > > For machines 2-5, free disk space is reported as 434.3 GB
> > > > > >
> > > > > > For at least the next day, I'm going to be trying to interpret my
> > > > > > results based on the existing Marvel documentation, but this anomaly jumped
> > > > > > out at me.
> > > > > > I don't know how the disk free space is read by Marvel, maybe it's
> > > > > > using a method that doesn't understand a VMware virtual disk? Is pretty
> > > > > > ordinary, formatted ext4. Maybe a system level dependency is missing on
> > > > > > these machines?
> > > > > >
> > > > > > Thx,
> > > > > > Tony
> > > > >
> > > > > --
> > > > > You received this message because you are subscribed to a topic in the
> > > > > Google Groups "elasticsearch" group.
> > > > > To unsubscribe from this topic, visit
> > > > > [https://groups.google.com/d/topic/elasticsearch/h5XbhxZCiIU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/h5XbhxZCiIU/unsubscribe).
> > > > > To unsubscribe from this group and all its topics, send an email to
> > > > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > > > > To view this discussion on the web visit
> > > > > [https://groups.google.com/d/msgid/elasticsearch/c608547a-ca43-4433-9af5-cde2143203e4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c608547a-ca43-4433-9af5-cde2143203e4%40googlegroups.com)
> > > > > .
> > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/134f1460-9605-4625-9779-b6a4355d81d7%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/134f1460-9605-4625-9779-b6a4355d81d7%40googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Boaz\_Leskes](https://discuss.elastic.co/u/Boaz_Leskes)
#### Post date: [January 31, 2014, 7:49pm UTC](https://discuss.elastic.co/t/marvel-reporting-incorrect-free-disk-space/15515/8 "2014-01-31T19:49:10Z")
Marvel measure the free space on the file system where the ES it monitors
keeps it's data. According to the node stats you sent, this
is /mnt/hgfs/DataShared/elasticsearch :
"data":[{"path":"/mnt/hgfs/DataShared/elasticsearch/nodes/0","mount":
"/mnt/hgfs","dev":".host:/","total":"904.3gb"
which is on a mount with a total size of 904GB.
Is this not correct?
On Friday, January 31, 2014 7:52:32 PM UTC+1, Tony Su wrote:
> Well, of course none of that is true...
> you include the mounted share which includes the raw data which
> is being input into the system.
>
> So, I guess that's where the anomaly exists...
> I doubt that the source where the raw data is coming from should be
> considered part of the "working environment" of Elasticsearch... I assume
> that "free space" should only include what ES should see as available to
> run, create and store (which of course is exclusive of the raw data source).
>
> But, that raises a curious observation...
> On Machine 1 (where all data is imported into an ES node, also running the
> LS shipper and indexers and Redis) space is probably being reported
> correctly (approx. half of the local disk, and although of course is using
> the same data mount share).
>
> Considering the above, I guess technically it might be considered that
> Machines 2-5 have access to the free disk space on the shared data mount
> although I can't imagine how it would be used in a normal setting. Maybe
> instead the ES paths should be parsed and only the partitions those paths
> point to should be reported as contributing to available "free space?" Or,
> maybe even more accurately only the ES Data Path partition should be
> reported? Taking this idea maybe a step further, there might be a question
> how to report the ES TMP path, if it's not tmpfs then it should also be
> included?
>
> Tony
>
> Tony
>
> On Friday, January 31, 2014 9:38:42 AM UTC-8, Boaz Leskes wrote:
>
> > Hi Tony,
> >
> > I'm a bit confused. These numbers seem to add up? Both marvel, df -h and
> > the ES stats say you have ~430GB free on ".host:/", where ES is storing
> > it's data.
> >
> > Boaz
> >
> > On Fri, Jan 31, 2014 at 5:55 PM, Tony Su [tony...@gmail.com](mailto:tony...@gmail.com) wrote:
> >
> > > Sorry, you're right. Typo in my curl command.
> > >
> > > corrected
> > >
> > > {"cluster\_name":"elasticsearch","nodes":{"8f-ZbebNT2e23DArE6WnIw":{"timestamp":1391187042684,"name":"ELASTICSEARCH-4","transport\_address":"inet[
> > > 192.168.248.172/192.168.248.172:9300
> > > ]","fs":{"timestamp":1391187042685,"total":{"total":"904.3gb","total\_in\_bytes":971057917952,"free":"431.7gb","free\_in\_bytes":463607828480,"available":"431.7gb","available\_in\_bytes":463607828480},"data":[{"path":"/mnt/hgfs/DataShared/elasticsearch/nodes/0","mount":"/mnt/hgfs","dev":".host:/","total":"904.3gb","total\_in\_bytes":971057917952,"free":"431.7gb","free\_in\_bytes":463607828480,"available":"431.7gb","available\_in\_bytes":463607828480}]}}}}
> > >
> > > On Friday, January 31, 2014 8:42:02 AM UTC-8, Boaz Leskes wrote:
> > >
> > > > Thx.
> > > >
> > > > Something went wrong with the curl command as it returned indices stats
> > > > only (which is the default). The "&fs" should have caused it to return file
> > > > system stats. Perhaps something went wrong with the '&'. Can you double
> > > > check?
> > > >
> > > > On Friday, January 31, 2014 5:10:02 PM UTC+1, Tony Su wrote:
> > > >
> > > > > SIGA is great if that is what you're using...
> > > > > If you'd like I can also test installing that separately to see if it
> > > > > makes a diff...
> > > > > Tony
> > > > >
> > > > > # curl -XGET "[http://localhost:9200/\_cluster/nodes/\_local/stats](http://localhost:9200/_cluster/nodes/_local/stats)?
> > > > >
> > > > > clear&fs"
> > > > > {"cluster\_name":"elasticsearch","nodes":{"8f-ZbebNT2e23DArE6WnIw":{"
> > > > > timestamp":1391183667580,"name":"ELASTICSEARCH-4","
> > > > > transport\_address":"inet[192.168.248.172/192.168.248.172:9300
> > > > > ]","indices":{"docs":{"count":2180,"deleted":0},"
> > > > > store":{"size":"1.5mb","size\_in\_bytes":1631986,"throttle\_
> > > > > time":"0s","throttle\_time\_in\_millis":0},"indexing":{"index\_
> > > > > total":0,"index\_time":"0s","index\_time\_in\_millis":0,"
> > > > > index\_current":0,"delete\_total":0,"delete\_time":"0s","
> > > > > delete\_time\_in\_millis":0,"delete\_current":0},"get":{"
> > > > > total":0,"get\_time":"0s","time\_in\_millis":0,"exists\_
> > > > > total":0,"exists\_time":"0s","exists\_time\_in\_millis":0,"
> > > > > missing\_total":0,"missing\_time":"0s","missing\_time\_in\_
> > > > > millis":0,"current":0},"search":{"open\_contexts":0,"
> > > > > query\_total":0,"query\_time":"0s","query\_time\_in\_millis":0,"
> > > > > query\_current":0,"fetch\_total":0,"fetch\_time":"0s","fetch\_
> > > > > time\_in\_millis":0,"fetch\_current":0}}}}}
> > > > >
> > > > > df -h
> > > > > Filesystem Size Used Avail Use% Mounted on
> > > > > /dev/sda2 15G 2.4G 12G 17% /
> > > > > devtmpfs 743M 32K 743M 1% /dev
> > > > > tmpfs 748M 0 748M 0% /dev/shm
> > > > > tmpfs 748M 3.1M 744M 1% /run
> > > > > tmpfs 748M 0 748M 0% /sys/fs/cgroup
> > > > > tmpfs 748M 3.1M 744M 1% /var/run
> > > > > tmpfs 748M 3.1M 744M 1% /var/lock
> > > > > .host:/ 905G 467G 438G 52% /mnt/hgfs
> > > > >
> > > > > On Friday, January 31, 2014 12:52:21 AM UTC-8, Boaz Leskes wrote:
> > > > >
> > > > > > Hi Tony,
> > > > > >
> > > > > > I suspect this is an issue with the reporting of the Nodes Stats API
> > > > > > (based on sigar). Can you please post the output of calling:
> > > > > >
> > > > > > curl -XGET "[http://localhost:9200/\_cluster/nodes/\_local/stats](http://localhost:9200/_cluster/nodes/_local/stats)?
> > > > > > clear&fs"
> > > > > >
> > > > > > &
> > > > > >
> > > > > > df -h
> > > > > >
> > > > > > on one of the machines 2-5?
> > > > > >
> > > > > > Thx,
> > > > > > Boaz
> > > > > >
> > > > > > On Thursday, January 30, 2014 11:39:48 PM UTC+1, Tony Su wrote:
> > > > > >
> > > > > > > Deployed in a VMware environment.
> > > > > > > Data is being consumed and is successful from source to ES
> > > > > > >
> > > > > > > source \> logstash shipper(and parser) \> redis \> logstash \> es cluster
> > > > > > >
> > > > > > > ES Cluster
> > > > > > > All openSUSE 13.1
> > > > > > > Machine 1 - 4GB ram - 16GB HDD - openSUSE/LXDE -ES -Logstash - netcat
> > > > > > > Machine 2 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
> > > > > > > Machine 3 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
> > > > > > > Machine 4 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
> > > > > > > Machine 5 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
> > > > > > >
> > > > > > > Marvel reports the disk space likely correctly only on the initial
> > > > > > > machine (8.8GB free)
> > > > > > > For machines 2-5, free disk space is reported as 434.3 GB
> > > > > > >
> > > > > > > For at least the next day, I'm going to be trying to interpret my
> > > > > > > results based on the existing Marvel documentation, but this anomaly jumped
> > > > > > > out at me.
> > > > > > > I don't know how the disk free space is read by Marvel, maybe it's
> > > > > > > using a method that doesn't understand a VMware virtual disk? Is pretty
> > > > > > > ordinary, formatted ext4. Maybe a system level dependency is missing on
> > > > > > > these machines?
> > > > > > >
> > > > > > > Thx,
> > > > > > > Tony
> > > > > >
> > > > > > --
> > > > > > You received this message because you are subscribed to a topic in the
> > > > > > Google Groups "elasticsearch" group.
> > > > > > To unsubscribe from this topic, visit
> > > > > > [https://groups.google.com/d/topic/elasticsearch/h5XbhxZCiIU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/h5XbhxZCiIU/unsubscribe).
> > > > > > To unsubscribe from this group and all its topics, send an email to
> > > > > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > > > > To view this discussion on the web visit
> > > > > > [https://groups.google.com/d/msgid/elasticsearch/c608547a-ca43-4433-9af5-cde2143203e4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c608547a-ca43-4433-9af5-cde2143203e4%40googlegroups.com)
> > > > > > .
> > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1cacd9ed-66e6-4133-92fa-fd6f22059dd3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1cacd9ed-66e6-4133-92fa-fd6f22059dd3%40googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Tony\_Su](https://discuss.elastic.co/u/Tony_Su)
#### Post date: [January 31, 2014, 10:54pm UTC](https://discuss.elastic.co/t/marvel-reporting-incorrect-free-disk-space/15515/9 "2014-01-31T22:54:14Z")
You're absolutely right.
I forgot I had changed that setting long ago when I was still feeling my
way around what are in these directories.
Also resolved some other weirdness I was seeing which raises an off
the beat question...
Because all my Machine2-4 data directories were pointing to a shared
location, they all were writing to, and reading from the same.
Besides the disk space anomaly I noticed, I also noticed that the IOps for
these machines were also zero, I assume because whatever metadata they
required was already created and available through a mounted share and
didn't require a network transfer.
Without testing, superficially it looks like the nodes seemed to
perform without error.
So, can a "shared metadata repository" be a normal and perhaps advantageous
ES configuration or is separate metadata stores for each node still
recommended? If it works, I speculate that any I/O bottlenecks should be
avoided but could potentially be a point of failure (single repository).
Thx,
Tony
On Friday, January 31, 2014 11:49:10 AM UTC-8, Boaz Leskes wrote:
> Marvel measure the free space on the file system where the ES it monitors
> keeps it's data. According to the node stats you sent, this
> is /mnt/hgfs/DataShared/elasticsearch :
>
> "data":[{"path":"/mnt/hgfs/DataShared/elasticsearch/nodes/0","mount":
> "/mnt/hgfs","dev":".host:/","total":"904.3gb"
>
> which is on a mount with a total size of 904GB.
>
> Is this not correct?
>
> On Friday, January 31, 2014 7:52:32 PM UTC+1, Tony Su wrote:
>
> > Well, of course none of that is true...
> > you include the mounted share which includes the raw data which
> > is being input into the system.
> >
> > So, I guess that's where the anomaly exists...
> > I doubt that the source where the raw data is coming from should be
> > considered part of the "working environment" of Elasticsearch... I assume
> > that "free space" should only include what ES should see as available to
> > run, create and store (which of course is exclusive of the raw data source).
> >
> > But, that raises a curious observation...
> > On Machine 1 (where all data is imported into an ES node, also running
> > the LS shipper and indexers and Redis) space is probably being reported
> > correctly (approx. half of the local disk, and although of course is using
> > the same data mount share).
> >
> > Considering the above, I guess technically it might be considered that
> > Machines 2-5 have access to the free disk space on the shared data mount
> > although I can't imagine how it would be used in a normal setting. Maybe
> > instead the ES paths should be parsed and only the partitions those paths
> > point to should be reported as contributing to available "free space?" Or,
> > maybe even more accurately only the ES Data Path partition should be
> > reported? Taking this idea maybe a step further, there might be a question
> > how to report the ES TMP path, if it's not tmpfs then it should also be
> > included?
> >
> > Tony
> >
> > Tony
> >
> > On Friday, January 31, 2014 9:38:42 AM UTC-8, Boaz Leskes wrote:
> >
> > > Hi Tony,
> > >
> > > I'm a bit confused. These numbers seem to add up? Both marvel, df -h and
> > > the ES stats say you have ~430GB free on ".host:/", where ES is storing
> > > it's data.
> > >
> > > Boaz
> > >
> > > On Fri, Jan 31, 2014 at 5:55 PM, Tony Su [tony...@gmail.com](mailto:tony...@gmail.com) wrote:
> > >
> > > > Sorry, you're right. Typo in my curl command.
> > > >
> > > > corrected
> > > >
> > > > {"cluster\_name":"elasticsearch","nodes":{"8f-ZbebNT2e23DArE6WnIw":{"timestamp":1391187042684,"name":"ELASTICSEARCH-4","transport\_address":"inet[
> > > > 192.168.248.172/192.168.248.172:9300
> > > > ]","fs":{"timestamp":1391187042685,"total":{"total":"904.3gb","total\_in\_bytes":971057917952,"free":"431.7gb","free\_in\_bytes":463607828480,"available":"431.7gb","available\_in\_bytes":463607828480},"data":[{"path":"/mnt/hgfs/DataShared/elasticsearch/nodes/0","mount":"/mnt/hgfs","dev":".host:/","total":"904.3gb","total\_in\_bytes":971057917952,"free":"431.7gb","free\_in\_bytes":463607828480,"available":"431.7gb","available\_in\_bytes":463607828480}]}}}}
> > > >
> > > > On Friday, January 31, 2014 8:42:02 AM UTC-8, Boaz Leskes wrote:
> > > >
> > > > > Thx.
> > > > >
> > > > > Something went wrong with the curl command as it returned indices
> > > > > stats only (which is the default). The "&fs" should have caused it to
> > > > > return file system stats. Perhaps something went wrong with the '&'. Can
> > > > > you double check?
> > > > >
> > > > > On Friday, January 31, 2014 5:10:02 PM UTC+1, Tony Su wrote:
> > > > >
> > > > > > SIGA is great if that is what you're using...
> > > > > > If you'd like I can also test installing that separately to see if it
> > > > > > makes a diff...
> > > > > > Tony
> > > > > >
> > > > > > # curl -XGET "[http://localhost:9200/\_cluster/nodes/\_local/stats](http://localhost:9200/_cluster/nodes/_local/stats)?
> > > > > >
> > > > > > clear&fs"
> > > > > > {"cluster\_name":"elasticsearch","nodes":{"8f-ZbebNT2e23DArE6WnIw":{"
> > > > > > timestamp":1391183667580,"name":"ELASTICSEARCH-4","
> > > > > > transport\_address":"inet[192.168.248.172/192.168.248.172:9300
> > > > > > ]","indices":{"docs":{"count":2180,"deleted":0},"
> > > > > > store":{"size":"1.5mb","size\_in\_bytes":1631986,"throttle\_
> > > > > > time":"0s","throttle\_time\_in\_millis":0},"indexing":{"index\_
> > > > > > total":0,"index\_time":"0s","index\_time\_in\_millis":0,"
> > > > > > index\_current":0,"delete\_total":0,"delete\_time":"0s","
> > > > > > delete\_time\_in\_millis":0,"delete\_current":0},"get":{"
> > > > > > total":0,"get\_time":"0s","time\_in\_millis":0,"exists\_
> > > > > > total":0,"exists\_time":"0s","exists\_time\_in\_millis":0,"
> > > > > > missing\_total":0,"missing\_time":"0s","missing\_time\_in\_
> > > > > > millis":0,"current":0},"search":{"open\_contexts":0,"
> > > > > > query\_total":0,"query\_time":"0s","query\_time\_in\_millis":0,"
> > > > > > query\_current":0,"fetch\_total":0,"fetch\_time":"0s","fetch\_
> > > > > > time\_in\_millis":0,"fetch\_current":0}}}}}
> > > > > >
> > > > > > df -h
> > > > > > Filesystem Size Used Avail Use% Mounted on
> > > > > > /dev/sda2 15G 2.4G 12G 17% /
> > > > > > devtmpfs 743M 32K 743M 1% /dev
> > > > > > tmpfs 748M 0 748M 0% /dev/shm
> > > > > > tmpfs 748M 3.1M 744M 1% /run
> > > > > > tmpfs 748M 0 748M 0% /sys/fs/cgroup
> > > > > > tmpfs 748M 3.1M 744M 1% /var/run
> > > > > > tmpfs 748M 3.1M 744M 1% /var/lock
> > > > > > .host:/ 905G 467G 438G 52% /mnt/hgfs
> > > > > >
> > > > > > On Friday, January 31, 2014 12:52:21 AM UTC-8, Boaz Leskes wrote:
> > > > > >
> > > > > > > Hi Tony,
> > > > > > >
> > > > > > > I suspect this is an issue with the reporting of the Nodes Stats API
> > > > > > > (based on sigar). Can you please post the output of calling:
> > > > > > >
> > > > > > > curl -XGET "[http://localhost:9200/\_cluster/nodes/\_local/stats](http://localhost:9200/_cluster/nodes/_local/stats)?
> > > > > > > clear&fs"
> > > > > > >
> > > > > > > &
> > > > > > >
> > > > > > > df -h
> > > > > > >
> > > > > > > on one of the machines 2-5?
> > > > > > >
> > > > > > > Thx,
> > > > > > > Boaz
> > > > > > >
> > > > > > > On Thursday, January 30, 2014 11:39:48 PM UTC+1, Tony Su wrote:
> > > > > > >
> > > > > > > > Deployed in a VMware environment.
> > > > > > > > Data is being consumed and is successful from source to ES
> > > > > > > >
> > > > > > > > source \> logstash shipper(and parser) \> redis \> logstash \> es
> > > > > > > > cluster
> > > > > > > >
> > > > > > > > ES Cluster
> > > > > > > > All openSUSE 13.1
> > > > > > > > Machine 1 - 4GB ram - 16GB HDD - openSUSE/LXDE -ES -Logstash -
> > > > > > > > netcat
> > > > > > > > Machine 2 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
> > > > > > > > Machine 3 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
> > > > > > > > Machine 4 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
> > > > > > > > Machine 5 - 1.5GB RAM -16GB HDD -openSUSE (no desktop) - ES
> > > > > > > >
> > > > > > > > Marvel reports the disk space likely correctly only on the initial
> > > > > > > > machine (8.8GB free)
> > > > > > > > For machines 2-5, free disk space is reported as 434.3 GB
> > > > > > > >
> > > > > > > > For at least the next day, I'm going to be trying to interpret my
> > > > > > > > results based on the existing Marvel documentation, but this anomaly jumped
> > > > > > > > out at me.
> > > > > > > > I don't know how the disk free space is read by Marvel, maybe it's
> > > > > > > > using a method that doesn't understand a VMware virtual disk? Is pretty
> > > > > > > > ordinary, formatted ext4. Maybe a system level dependency is missing on
> > > > > > > > these machines?
> > > > > > > >
> > > > > > > > Thx,
> > > > > > > > Tony
> > > > > > >
> > > > > > > --
> > > > > > > You received this message because you are subscribed to a topic in the
> > > > > > > Google Groups "elasticsearch" group.
> > > > > > > To unsubscribe from this topic, visit
> > > > > > > [https://groups.google.com/d/topic/elasticsearch/h5XbhxZCiIU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/h5XbhxZCiIU/unsubscribe)
> > > > > > > .
> > > > > > > To unsubscribe from this group and all its topics, send an email to
> > > > > > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > > > > > To view this discussion on the web visit
> > > > > > > [https://groups.google.com/d/msgid/elasticsearch/c608547a-ca43-4433-9af5-cde2143203e4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c608547a-ca43-4433-9af5-cde2143203e4%40googlegroups.com)
> > > > > > > .
> > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/594a220c-a9c8-4359-b40f-759171acbf02%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/594a220c-a9c8-4359-b40f-759171acbf02%40googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 1:53am UTC](https://discuss.elastic.co/t/marvel-reporting-incorrect-free-disk-space/15515/10 "2017-07-06T01:53:30Z")