Metricbeat with AWS EFS

[Joey_Coleman]

Hey, has anyone configured metricbeat on a system with a AWS EFS share mounted? Because AWS shows that the volume has 9223372036854775808 bytes (8 Exabytes) free, I'm getting warnings like below showing up in the log.

2017-02-21T19:28:32Z WARN Can not index event (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse [system.filesystem.total]","caused_by":{"type":"json_parse_exception","reason":"Numeric value (9223372036854775808) out of range of long (-9223372036854775808 - 9223372036854775807)\n at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@6e55ae3; line: 1, column: 398]"}}
2017-02-21T19:28:32Z WARN Can not index event (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse [system.fsstat.total_size.free]","caused_by":{"type":"json_parse_exception","reason":"Numeric value (9223372051157159936) out of range of long (-9223372036854775808 - 9223372036854775807)\n at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@21e3b3e8; line: 1, column: 269]"}}

The relevant subsection of the config is:

metricbeat.modules:
- module: system
  metricsets:
    - filesystem
    - fsstat
  enabled: true
  period: 30s

It is possible to use filtering to exclude events from the filesystem metricset, but the fsstat produces aggregate values, so will include the EFS mount regardless.

Steps to reproduce:

1. Instantiate a linux box in AWS
2. Create an AWS EFS share
3. Mount the EFS share on the linux box
4. Start Metricbeat, watch the logs

Any suggestions on how to handle situations where a system might have more that 2^63 bytes of storage available?

—Joey

[andrewkroh]

I'm curious what stat -f and df show for that EFS volume?

This is kind of similar to Failed to parse [system.filesystem.files]: Numeric value (18446744073709551615) out of range of long but in this case it's for a huge filesystem instead of a invalid value. So I think we need to come up with some way to handle this.

[Joey_Coleman]

Easy to produce. From stat you can see that AWS provides exactly 2^63 bytes of storage (2^43 1MB blocks).

$ df /mnt/efs
Filesystem                                       1K-blocks     Used        Available Use% Mounted on
fs-xxxxxxxx.efs.us-east-1.amazonaws.com:/ 9007199254740992 19168256 9007199235572736   1% /mnt/efs

$ df -h /mnt/efs
Filesystem                                 Size  Used Avail Use% Mounted on
fs-xxxxxxxx.efs.us-east-1.amazonaws.com:/  8.0E   19G  8.0E   1% /mnt/efs

$ stat -f /mnt/efs
  File: "/mnt/efs"
    ID: 0        Namelen: 255     Type: nfs
Block size: 1048576    Fundamental block size: 1048576
Blocks: Total: 8796093022208 Free: 8796093003489 Available: 8796093003489
Inodes: Total: 0          Free: 0

[andrewkroh]

@Joey_Coleman Would you mind opening a bug report on Github for this so we don't lose track of it?

[Joey_Coleman]

Happy to — didn't initially as the guidelines suggested posting on the boards first

[Joey_Coleman]

Opened as https://github.com/elastic/beats/issues/3687

[ruflin]

Thanks @Joey_Coleman. We really appreciate you posted it first here for discussions.