Metricbeat fsstat values are too high

Elastic Stack Beats
1

Hi, I'm new to the ELK stack and enjoying it so far. I'm running metricbeat on multiple servers and the fsstat.total_size.total value is correct on all of them except for one server which happens to be the only one running docker (not sure if that is relevant). This server has a 100GB disk, but it is reported as 945GB by metricbeat.

vkurup@XXXX06:~$ df --total
Filesystem                  1K-blocks     Used Available Use% Mounted on
udev                          1996052        0   1996052   0% /dev
tmpfs                          409604     6280    403324   2% /run
/dev/mapper/XXXX06--vg-root  98326140 25414364  67893972  28% /
tmpfs                         2048004     1292   2046712   1% /dev/shm
tmpfs                            5120        0      5120   0% /run/lock
tmpfs                         2048004        0   2048004   0% /sys/fs/cgroup
/dev/xvda1                     482922   107898    350090  24% /boot
tmpfs                          409604        0    409604   0% /run/user/999
tmpfs                          409604        0    409604   0% /run/user/1000
total                       106135054 25529834  75562482  26% -

JSON output from metricbeat:

{
  "_index": "metricbeat-2017.11.13",
  "_type": "metricsets",
  "_id": "AV-22MFyewUwG2sGb8lj",
  "_version": 1,
  "_score": null,
  "_source": {
    "system": {
      "fsstat": {
        "total_size": {
          "total": 1015392872448,
          "used": 259980732416,
          "free": 755412140032
        },
        "count": 56,
        "total_files": 70378835
      }
    },
    "@timestamp": "2017-11-13T19:26:18.555Z",
    "beat": {
      "name": "XXXX06",
      "hostname": "XXXX06",
      "version": "5.6.4"
    },
    "@version": "1",
    "host": "XXXX06",
    "metricset": {
      "name": "fsstat",
      "rtt": 1574,
      "module": "system"
    },
    "fields": {
      "environment": "production",
      "roles": [
        "sentry",
        "logs-server"
      ]
    },
    "type": "metricsets",
    "tags": [
      "beats_input_raw_event"
    ]
  },
  "fields": {
    "@timestamp": [
      1510601178555
    ]
  },
  "highlight": {
    "host": [
      "@kibana-highlighted-field@XXXX06@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1510601178555
  ]
}

So the value reported by metricbeat appears way too high. How can I go about debugging this?

Thank you!

2

Any chance you could enable the filesystem metricset and check the output there. I'm assuming there are some values which are not reported by df

3

I don't know the best way to share that data here. I enabled the filesystem metric and took a screenshot (filtered to items with a positive value for filesystem.available:

image
image.png1398×851 111 KB

So, this is showing 10 entries with an available size of 60GB each. Nine of those are docker entries, and one is my host device. If fsstat is counting those docker entries, then it makes sense why it's reporting a value 10x the size of my actual disk.

I tried adding this to my config:

  filters:
    # don't double-count docker file systems
    - drop_event.when.regexp.mount_point: '^/var/lib/docker/aufs'

That makes the extra entries in the filesystem metrics go away, but it doesn't affect the fsstat metrics.

4

Sorry for the late reply. What is the type of the docker file systems? If it has it's own type you could use ignore_type: https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-metricset-system-filesystem.html#_configuration_5

Unfortunately the regexp above does not apply to the fsstat metrics as it's only applied after the event is generated.

5

Thanks for the response!

Unfortunately, the type is 'ext4' for the /var/lib/docker/aufs entry and '-' for all of the 'mnt' subdirectories underneath. Oh well... my drop_event filter works for the filesystem module.

I guess the next step is to create a dashboard that manually calculates and displays stats based on the filesystem reports, rather than using the fsstat module? That may be above my current ability level, but I'll take a look. :grinning: Pointers to docs appreciated!

6

Yes, you should be able to use the sum aggregation for that: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-sum-aggregation.html

Could you also open an issue for the above on Github? I'm assuming you will not be the only one hitting this isssue with docker and it would be nice to have a way to filter these volumes out from fsstat.

7

Thanks for the pointer! I've created a Github issue. Let me know if I should add additional info there.

8

Thanks, lets continue the conversation htere.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.