Failed to parse [system.filesystem.files]: Numeric value (18446744073709551615) out of range of long

[max0x7ba (Maxim Egorushkin)]

Elasticsearch version:
elasticsearch-5.0.0
metricbeat-5.0.1

Plugins installed:
None

JVM version:
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)

OS version:
Red Hat Enterprise Linux Server release 6.8 (Santiago) x86_64

Description of the problem including expected versus actual behavior:
Metricbeat sends system.filesystem.files field with value of 18446744073709551615, which is (2^64 - 1).

Steps to reproduce:

1. Enable metricset filesystem in metricbeat.modules -> module: system,
2. Start metricbeat.
3. Observe the warning in the log.

Provide logs (if relevant):
2016-11-22T05:35:17-05:00 WARN Can not index event (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse [system.filesystem.files]","caused_by":{"type":"json_parse_exception","reason":"Numeric value (18446744073709551615) out of range of long (-9223372036854775808 - 9223372036854775807)\n at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@39fece1e; line: 1, column: 234]"}}

[andrewkroh]

What's the filesystem type for that mount point?

It looks like the filesystem is reporting -1 for the total number of nodes. The field is defined as being unsigned so a -1 is interpreted incorrectly by things that read that value. (I have seen this before with cephfs.)

What does stat -f <mount_point> show for that filesystem?

[max0x7ba (Maxim Egorushkin)]

This is /dev/hugepages what is causing troubles:

$ df -h
[...]
nodev                  16G  688M   16G   5% /dev/hugepages
[...]

$ stat -f /dev/hugepages
  File: "/dev/hugepages"
    ID: 0        Namelen: 255     Type: hugetlbfs
Block size: 2097152    Fundamental block size: 2097152
Blocks: Total: 8192       Free: 7848       Available: 7848
Inodes: Total: 18446744073709551615 Free: -1

[andrewkroh]

One way you can workaround this issue is to filter out events coming from that mount point. There's a good example in the documentation, see https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-metricset-system-filesystem.html#_filtering. Most of the mount points in that example are not useful to report on so this is a good option IMO.

You could also setup a filter with a condition like when system.filesystem.files == 18446744073709551615, drop_event. Or your could just drop the system.filesystem.files field.

Processor Docs

[max0x7ba (Maxim Egorushkin)]

I see, thank you.