Metricbeat 5.3.0 reporting strange memory value

[michbsd]

beats 5.3.0 on FreeBSD

linprocfs has been mounted

and this has been working for months.

This morning I noticed errors in my elasticsearch log, something like this:

"[2018-03-20T15:53:19,197][DEBUG][o.e.a.b.TransportShardBulkAction] [7xHqegG] [metricbeat-2018.03.20][4] failed to execute bulk item (index) BulkShardRequest [[metricbeat-2018.03.20][4]] containing [3] requests
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [system.memory.actual.used.bytes]"
[...]
Caused by: com.fasterxml.jackson.core.JsonParseException: Numeric value (18446744073594953728) out of range of long (-9223372036854775808 - 9223372036854775807)
 at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@36de2852; line: 1, column: 273]

I identified the offending beat host, sending this:

  "system": {
    "memory": {
      "actual": {
        "free": 137388007424,
        "used": {
          "bytes": 18446744073624211456,
          "pct": 134350952.106100
        }
      },
      "free": 137135058944,
      "swap": {
        "free": 2139643904,
        "total": 2147479552,
        "used": {
          "bytes": 7835648,
          "pct": 0.003600
        }
      },
      "total": 137302667264,
      "used": {
        "bytes": 167608320,
        "pct": 0.001200
      }
    }
  },
  "type": "metricsets"
}

bytes": 18446744073624211456, - that's a lot of used bytes..

Here's the meminfo output:
❯ cat /compat/linux/proc/meminfo [5:45:00 PM]
total: used: free: shared: buffers: cached:
Mem: 137302667264 167362560 137135304704 14020608 0 252948480
Swap: 2147479552 7835648 2139643904
MemTotal: 134084636 kB
MemFree: 133921196 kB
MemShared: 13692 kB
Buffers: 0 kB
Cached: 247020 kB
SwapTotal: 2097148 kB
SwapFree: 2089496 kB

I have update to gosigar 0.9.0

Any ideas?

[jsoriano (Jaime Soriano)]

Did you only update gosigar? What version were you using before?

[jsoriano (Jaime Soriano)]

I have created an issue in gosigar to investigate this (https://github.com/elastic/gosigar/issues/102). It seems that in some cases actual free memory is greater than total memory, and this is used to calculate used memory.

[michbsd]

Not entirely sure. Whatever was released with 5.3.0 of beats..

I disabled memory metricset for a few days, and reenabled it again, and the error was gone.

In the meantime I have upgraded to beats 5.6.8 - and the error is gone. Very strange though.

[michbsd]

Great - will follow the bug analysis

[michbsd]

Spoke to fast- I am now seeing this error from two different hosts running metricbeat 5.6.8