# Metricbeat fsstat values are too high

**URL:** <https://discuss.elastic.co/t/metricbeat-fsstat-values-are-too-high/107453>\
**Category:** Beats\
**Tags:** metricbeat\
**Created:** [November 13, 2017, 7:47pm UTC](https://discuss.elastic.co/t/metricbeat-fsstat-values-are-too-high/107453 "2017-11-13T19:47:14Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![vkurup](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vkurup/32/24159_2.png) [@vkurup](https://discuss.elastic.co/u/vkurup)\
**Post date:** [November 13, 2017, 7:47pm UTC](https://discuss.elastic.co/t/metricbeat-fsstat-values-are-too-high/107453/1 "2017-11-13T19:47:14Z")

</div>

Hi, I'm new to the ELK stack and enjoying it so far. I'm running metricbeat on multiple servers and the fsstat.total\_size.total value is correct on all of them except for one server which happens to be the only one running docker (not sure if that is relevant). This server has a 100GB disk, but it is reported as 945GB by metricbeat.

```
vkurup@XXXX06:~$ df --total
Filesystem 1K-blocks Used Available Use% Mounted on
udev 1996052 0 1996052 0% /dev
tmpfs 409604 6280 403324 2% /run
/dev/mapper/XXXX06--vg-root 98326140 25414364 67893972 28% /
tmpfs 2048004 1292 2046712 1% /dev/shm
tmpfs 5120 0 5120 0% /run/lock
tmpfs 2048004 0 2048004 0% /sys/fs/cgroup
/dev/xvda1 482922 107898 350090 24% /boot
tmpfs 409604 0 409604 0% /run/user/999
tmpfs 409604 0 409604 0% /run/user/1000
total 106135054 25529834 75562482 26% -

```

JSON output from metricbeat:

```
{
  "_index": "metricbeat-2017.11.13",
  "_type": "metricsets",
  "_id": "AV-22MFyewUwG2sGb8lj",
  "_version": 1,
  "_score": null,
  "_source": {
    "system": {
      "fsstat": {
        "total_size": {
          "total": 1015392872448,
          "used": 259980732416,
          "free": 755412140032
        },
        "count": 56,
        "total_files": 70378835
      }
    },
    "@timestamp": "2017-11-13T19:26:18.555Z",
    "beat": {
      "name": "XXXX06",
      "hostname": "XXXX06",
      "version": "5.6.4"
    },
    "@version": "1",
    "host": "XXXX06",
    "metricset": {
      "name": "fsstat",
      "rtt": 1574,
      "module": "system"
    },
    "fields": {
      "environment": "production",
      "roles": [
        "sentry",
        "logs-server"
      ]
    },
    "type": "metricsets",
    "tags": [
      "beats_input_raw_event"
    ]
  },
  "fields": {
    "@timestamp": [
      1510601178555
    ]
  },
  "highlight": {
    "host": [
      "@kibana-highlighted-field@XXXX06@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1510601178555
  ]
}

```

So the value reported by metricbeat appears way too high. How can I go about debugging this?

Thank you!

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [November 14, 2017, 11:16pm UTC](https://discuss.elastic.co/t/metricbeat-fsstat-values-are-too-high/107453/2 "2017-11-14T23:16:37Z")

</div>

Any chance you could enable the `filesystem` metricset and check the output there. I'm assuming there are some values which are not reported by `df`

---

<div class="post-metadata">

**Author:** ![vkurup](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vkurup/32/24159_2.png) [@vkurup](https://discuss.elastic.co/u/vkurup)\
**Post date:** [November 15, 2017, 12:28am UTC](https://discuss.elastic.co/t/metricbeat-fsstat-values-are-too-high/107453/3 "2017-11-15T00:28:52Z")

</div>

I don't know the best way to share that data here. I enabled the filesystem metric and took a screenshot (filtered to items with a positive value for filesystem.available:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/e/deb34a2b6c7a3ef59053ea00635bcfd847c29114.png)

So, this is showing 10 entries with an available size of 60GB each. Nine of those are docker entries, and one is my host device. If fsstat is counting those docker entries, then it makes sense why it's reporting a value 10x the size of my actual disk.

I tried adding this to my config:

```
  filters:
    # don't double-count docker file systems
    - drop_event.when.regexp.mount_point: '^/var/lib/docker/aufs'

```

That makes the extra entries in the filesystem metrics go away, but it doesn't affect the fsstat metrics.

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [November 20, 2017, 4:05am UTC](https://discuss.elastic.co/t/metricbeat-fsstat-values-are-too-high/107453/4 "2017-11-20T04:05:30Z")

</div>

Sorry for the late reply. What is the `type` of the docker file systems? If it has it's own type you could use `ignore_type`: [https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-metricset-system-filesystem.html#\_configuration\_5](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-metricset-system-filesystem.html#_configuration_5)

Unfortunately the regexp above does not apply to the fsstat metrics as it's only applied after the event is generated.

---

<div class="post-metadata">

**Author:** ![vkurup](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vkurup/32/24159_2.png) [@vkurup](https://discuss.elastic.co/u/vkurup)\
**Post date:** [November 28, 2017, 3:54pm UTC](https://discuss.elastic.co/t/metricbeat-fsstat-values-are-too-high/107453/5 "2017-11-28T15:54:54Z")

</div>

Thanks for the response!

Unfortunately, the type is 'ext4' for the /var/lib/docker/aufs entry and '-' for all of the 'mnt' subdirectories underneath. Oh well... my `drop_event` filter works for the filesystem module.

I guess the next step is to create a dashboard that manually calculates and displays stats based on the filesystem reports, rather than using the fsstat module? That may be above my current ability level, but I'll take a look. 😀 Pointers to docs appreciated!

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [November 29, 2017, 2:51am UTC](https://discuss.elastic.co/t/metricbeat-fsstat-values-are-too-high/107453/6 "2017-11-29T02:51:25Z")

</div>

Yes, you should be able to use the sum aggregation for that: [https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-sum-aggregation.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-sum-aggregation.html)

Could you also open an issue for the above on Github? I'm assuming you will not be the only one hitting this isssue with docker and it would be nice to have a way to filter these volumes out from fsstat.

---

<div class="post-metadata">

**Author:** ![vkurup](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vkurup/32/24159_2.png) [@vkurup](https://discuss.elastic.co/u/vkurup)\
**Post date:** [November 30, 2017, 2:10am UTC](https://discuss.elastic.co/t/metricbeat-fsstat-values-are-too-high/107453/7 "2017-11-30T02:10:24Z")

</div>

Thanks for the pointer! I've created a [Github issue](https://github.com/elastic/beats/issues/5767). Let me know if I should add additional info there.

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [November 30, 2017, 11:37pm UTC](https://discuss.elastic.co/t/metricbeat-fsstat-values-are-too-high/107453/8 "2017-11-30T23:37:31Z")

</div>

Thanks, lets continue the conversation htere.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 28, 2017, 11:38pm UTC](https://discuss.elastic.co/t/metricbeat-fsstat-values-are-too-high/107453/9 "2017-12-28T23:38:10Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.