Failed to publish events caused by: unsupported float value: NaN

Elastic Stack Beats
1

Metricbeat version: 6.3.1
OS: CentOS 7.1
Config:

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
  #reload.period: 10s
setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
setup.dashboards.enabled: true
setup.kibana:
  host: "localhost:5601"
  username: "xxx"
  password: "xxx"
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]

system.yml:

- module: system
  period: 1m
  metricsets:
    - filesystem
    - fsstat
  processors:
  - drop_event.when.regexp:
      system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)'

- module: system
  period: 15m
  metricsets:
    - uptime

#- module: system
#  period: 5m
#  metricsets:
#    - raid
#  raid.mount_point: '/'

- module: system
  period: 10s
  metricsets:
    - cpu
    - core
    - load
    - memory
    - network         # Network IO
    - process         # Per process metrics
    - process_summary # Process summary
    - diskio          # Disk IO
  cpu.metrics:  ["percentages"]  # The other available options are normalized_percentages and ticks.
  core.metrics: ["percentages"]  # The other available option is ticks.
  process.include_top_n:
    by_cpu: 5      # include top 5 processes by CPU
    by_memory: 5   # include top 5 processes by memory

The logs is full of

2018-07-21T15:36:48.960+0800    ERROR   logstash/async.go:235   Failed to publish events caused by: unsupported float value: NaN
2018-07-21T15:36:49.960+0800    ERROR   pipeline/output.go:92   Failed to publish events: unsupported float value: NaN

No other useful debug info. Any thoughts? Thanks.

2

Could you narrow down which metricset is returning incorrect events?
Could you please share the output of ./metricbeat -e -d "*"?

3

Hi kvch, thanks for the reply.

I have only two modules enabled, docker and system. When I disable system (by running mv /etc/metricbeat/modules.d/docker.yml /etc/metricbeat/modules.d/docker.yml.disabled), the error message no longer appears. So it should be a problem within system module.

This is the output of ./metricbeat -e -d "*":

2018-07-23T22:53:20.940+0800    DEBUG   [logstash]      logstash/async.go:142   123 events out of 123 events sent to logstash host localhost:5044. Continue sending
2018-07-23T22:53:20.940+0800    DEBUG   [logstash]      logstash/async.go:99    close connection
2018-07-23T22:53:20.940+0800    DEBUG   [transport]     transport/client.go:114 closing
2018-07-23T22:53:20.940+0800    ERROR   logstash/async.go:235   Failed to publish events caused by: unsupported float value: NaN
2018-07-23T22:53:20.940+0800    DEBUG   [logstash]      logstash/async.go:99    close connection
2018-07-23T22:53:21.941+0800    ERROR   pipeline/output.go:92   Failed to publish events: unsupported float value: NaN
2018-07-23T22:53:20.940+0800    DEBUG   [logstash]      logstash/async.go:99    close connection                                                                        [144/1850]
2018-07-23T22:53:21.941+0800    ERROR   pipeline/output.go:92   Failed to publish events: unsupported float value: NaN
2018-07-23T22:53:21.941+0800    DEBUG   [logstash]      logstash/async.go:94    connect
2018-07-23T22:53:21.942+0800    DEBUG   [logstash]      logstash/async.go:142   123 events out of 123 events sent to logstash host localhost:5044. Continue sending
2018-07-23T22:53:21.942+0800    DEBUG   [logstash]      logstash/async.go:99    close connection
2018-07-23T22:53:21.942+0800    DEBUG   [transport]     transport/client.go:114 closing
2018-07-23T22:53:21.942+0800    ERROR   logstash/async.go:235   Failed to publish events caused by: unsupported float value: NaN
2018-07-23T22:53:21.942+0800    DEBUG   [logstash]      logstash/async.go:99    close connection
2018-07-23T22:53:22.942+0800    ERROR   pipeline/output.go:92   Failed to publish events: unsupported float value: NaN
2018-07-23T22:53:22.942+0800    DEBUG   [logstash]      logstash/async.go:94    connect
2018-07-23T22:53:22.944+0800    DEBUG   [logstash]      logstash/async.go:142   123 events out of 123 events sent to logstash host localhost:5044. Continue sending
2018-07-23T22:53:22.944+0800    DEBUG   [logstash]      logstash/async.go:99    close connection
2018-07-23T22:53:22.944+0800    DEBUG   [transport]     transport/client.go:114 closing
2018-07-23T22:53:22.944+0800    ERROR   logstash/async.go:235   Failed to publish events caused by: unsupported float value: NaN
2018-07-23T22:53:22.944+0800    DEBUG   [logstash]      logstash/async.go:99    close connection
2018-07-23T22:53:23.944+0800    ERROR   pipeline/output.go:92   Failed to publish events: unsupported float value: NaN
2018-07-23T22:53:23.944+0800    DEBUG   [logstash]      logstash/async.go:94    connect
2018-07-23T22:53:23.945+0800    DEBUG   [logstash]      logstash/async.go:142   123 events out of 123 events sent to logstash host localhost:5044. Continue sending
2018-07-23T22:53:23.945+0800    DEBUG   [logstash]      logstash/async.go:99    close connection
2018-07-23T22:53:23.945+0800    DEBUG   [transport]     transport/client.go:114 closing
2018-07-23T22:53:23.946+0800    ERROR   logstash/async.go:235   Failed to publish events caused by: unsupported float value: NaN
2018-07-23T22:53:23.946+0800    DEBUG   [logstash]      logstash/async.go:99    close connection
2018-07-23T22:53:23.946+0800    INFO    [publish]       pipeline/retry.go:138   Drop batch
2018-07-23T22:53:23.946+0800    DEBUG   [memqueue]      memqueue/ackloop.go:143 ackloop: receive ack [1: 0, 123]
2018-07-23T22:53:23.946+0800    DEBUG   [memqueue]      memqueue/eventloop.go:518       broker ACK events: count=123, start-seq=32, end-seq=154
2018-07-23T22:53:23.946+0800    DEBUG   [memqueue]      memqueue/ackloop.go:111 ackloop: return ack to broker loop:123
2018-07-23T22:53:23.946+0800    DEBUG   [memqueue]      memqueue/ackloop.go:114 ackloop:  done send ack
4

Hi - could you please help? No clue what to do next. Thanks

5

Great! And which metricset of system produces this error?