Failed to encode event: unsupported float value: NaN

[alexus]

Hello World!

I'm using Elastic stack 6.6.0 and I'm seeing following message in: /var/log/metricbeat/metricbeat

2019-02-16T04:20:27.500Z ERROR elasticsearch/client.go:376 Failed to encode event: unsupported float value: NaN

• configuration block:
  • type: output
  • output type: elasticsearch
  • hosts: localhost:9200
  • username: elastic
  • password: ********

[Kaiyan_Sheng]

Hi @alexus, I actually saw the same error today What module are you running? If I can reproduce it on my side, that would be very helpful to debug.

[steffens (Steffen Siering)]

Related github issue: https://github.com/elastic/beats/issues/9212

[alexus]

I believe that's system module as I'm trying to get Infra going and have docker, system and output modules enabled...

worth to note, without beats management everything works as expected, with however it works for a little bit, then crashes, until one restarts everything and then it works again (for some time).

[steffens (Steffen Siering)]

worth to note, without beats management everything works as expected, with however it works for a little bit, then crashes, until one restarts everything and then it works again (for some time).

Hm, that's interesting. Can you share configuration details and logs?

[alexus]

config: I didn't modified any of these configuration files, as I use central management instead.

# grep -v ^# /etc/metricbeat/metricbeat.yml
    management:
      enabled: true
      period: 1m0s
      access_token: ${management.accesstoken}
      kibana:
        protocol: https
        host: X:443
        ssl: null
        timeout: 10s
        ignoreversion: true
      blacklist:
        output: console|file
    # 

&

    # cat /etc/metricbeat/modules.d/system.yml 
    # Module: system
    # Docs: https://www.elastic.co/guide/en/beats/metricbeat/6.6/metricbeat-module-system.html

    - module: system
      period: 10s
      metricsets:
        - cpu
        - load
        - memory
        - network
        - process
        - process_summary
        #- core
        #- diskio
        #- socket
      process.include_top_n:
        by_cpu: 5      # include top 5 processes by CPU
        by_memory: 5   # include top 5 processes by memory

    - module: system
      period: 1m
      metricsets:
        - filesystem
        - fsstat
      processors:
      - drop_event.when.regexp:
          system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)'

    - module: system
      period: 15m
      metricsets:
        - uptime

    #- module: system
    #  period: 5m
    #  metricsets:
    #    - raid
    #  raid.mount_point: '/'
    #

/var/log/metricbeat/metricbeat:

2019-02-20T21:16:51.014Z	ERROR	elasticsearch/client.go:376	Failed to encode event: unsupported float value: NaN
INFO	[monitoring]	log/log.go:144	Non-zero metrics in the last 30s

a lot of ERRORs and now and then INFO (monitoring).

[steffens (Steffen Siering)]

Which modules have you configured via Central Management.

If you enable the debug selector elasticsearch metricbeat will print a debug messages with the original event as is right after the error. This might help us in identifying the module the event did originate from.

[alexus]

The end goal is: Get up and running | Infrastructure Monitoring Guide [6.6] | Elastic.

---

I followed Configure logging, looks like it's Docker module!