Failed to publish events caused by: unsupported float value: NaN

Metricbeat version: 6.3.1
OS: CentOS 7.1
Config:

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
  #reload.period: 10s
setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
setup.dashboards.enabled: true
setup.kibana:
  host: "localhost:5601"
  username: "xxx"
  password: "xxx"
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]

system.yml:

- module: system
  period: 1m
  metricsets:
    - filesystem
    - fsstat
  processors:
  - drop_event.when.regexp:
      system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)'

- module: system
  period: 15m
  metricsets:
    - uptime

#- module: system
#  period: 5m
#  metricsets:
#    - raid
#  raid.mount_point: '/'

- module: system
  period: 10s
  metricsets:
    - cpu
    - core
    - load
    - memory
    - network         # Network IO
    - process         # Per process metrics
    - process_summary # Process summary
    - diskio          # Disk IO
  cpu.metrics:  ["percentages"]  # The other available options are normalized_percentages and ticks.
  core.metrics: ["percentages"]  # The other available option is ticks.
  process.include_top_n:
    by_cpu: 5      # include top 5 processes by CPU
    by_memory: 5   # include top 5 processes by memory

The logs is full of

2018-07-21T15:36:48.960+0800    ERROR   logstash/async.go:235   Failed to publish events caused by: unsupported float value: NaN
2018-07-21T15:36:49.960+0800    ERROR   pipeline/output.go:92   Failed to publish events: unsupported float value: NaN

No other useful debug info. Any thoughts? Thanks.

Could you narrow down which metricset is returning incorrect events?
Could you please share the output of ./metricbeat -e -d "*"?

Hi kvch, thanks for the reply.

I have only two modules enabled, docker and system. When I disable system (by running mv /etc/metricbeat/modules.d/docker.yml /etc/metricbeat/modules.d/docker.yml.disabled), the error message no longer appears. So it should be a problem within system module.

This is the output of ./metricbeat -e -d "*":

2018-07-23T22:53:20.940+0800    DEBUG   [logstash]      logstash/async.go:142   123 events out of 123 events sent to logstash host localhost:5044. Continue sending
2018-07-23T22:53:20.940+0800    DEBUG   [logstash]      logstash/async.go:99    close connection
2018-07-23T22:53:20.940+0800    DEBUG   [transport]     transport/client.go:114 closing
2018-07-23T22:53:20.940+0800    ERROR   logstash/async.go:235   Failed to publish events caused by: unsupported float value: NaN
2018-07-23T22:53:20.940+0800    DEBUG   [logstash]      logstash/async.go:99    close connection
2018-07-23T22:53:21.941+0800    ERROR   pipeline/output.go:92   Failed to publish events: unsupported float value: NaN
2018-07-23T22:53:20.940+0800    DEBUG   [logstash]      logstash/async.go:99    close connection                                                                        [144/1850]
2018-07-23T22:53:21.941+0800    ERROR   pipeline/output.go:92   Failed to publish events: unsupported float value: NaN
2018-07-23T22:53:21.941+0800    DEBUG   [logstash]      logstash/async.go:94    connect
2018-07-23T22:53:21.942+0800    DEBUG   [logstash]      logstash/async.go:142   123 events out of 123 events sent to logstash host localhost:5044. Continue sending
2018-07-23T22:53:21.942+0800    DEBUG   [logstash]      logstash/async.go:99    close connection
2018-07-23T22:53:21.942+0800    DEBUG   [transport]     transport/client.go:114 closing
2018-07-23T22:53:21.942+0800    ERROR   logstash/async.go:235   Failed to publish events caused by: unsupported float value: NaN
2018-07-23T22:53:21.942+0800    DEBUG   [logstash]      logstash/async.go:99    close connection
2018-07-23T22:53:22.942+0800    ERROR   pipeline/output.go:92   Failed to publish events: unsupported float value: NaN
2018-07-23T22:53:22.942+0800    DEBUG   [logstash]      logstash/async.go:94    connect
2018-07-23T22:53:22.944+0800    DEBUG   [logstash]      logstash/async.go:142   123 events out of 123 events sent to logstash host localhost:5044. Continue sending
2018-07-23T22:53:22.944+0800    DEBUG   [logstash]      logstash/async.go:99    close connection
2018-07-23T22:53:22.944+0800    DEBUG   [transport]     transport/client.go:114 closing
2018-07-23T22:53:22.944+0800    ERROR   logstash/async.go:235   Failed to publish events caused by: unsupported float value: NaN
2018-07-23T22:53:22.944+0800    DEBUG   [logstash]      logstash/async.go:99    close connection
2018-07-23T22:53:23.944+0800    ERROR   pipeline/output.go:92   Failed to publish events: unsupported float value: NaN
2018-07-23T22:53:23.944+0800    DEBUG   [logstash]      logstash/async.go:94    connect
2018-07-23T22:53:23.945+0800    DEBUG   [logstash]      logstash/async.go:142   123 events out of 123 events sent to logstash host localhost:5044. Continue sending
2018-07-23T22:53:23.945+0800    DEBUG   [logstash]      logstash/async.go:99    close connection
2018-07-23T22:53:23.945+0800    DEBUG   [transport]     transport/client.go:114 closing
2018-07-23T22:53:23.946+0800    ERROR   logstash/async.go:235   Failed to publish events caused by: unsupported float value: NaN
2018-07-23T22:53:23.946+0800    DEBUG   [logstash]      logstash/async.go:99    close connection
2018-07-23T22:53:23.946+0800    INFO    [publish]       pipeline/retry.go:138   Drop batch
2018-07-23T22:53:23.946+0800    DEBUG   [memqueue]      memqueue/ackloop.go:143 ackloop: receive ack [1: 0, 123]
2018-07-23T22:53:23.946+0800    DEBUG   [memqueue]      memqueue/eventloop.go:518       broker ACK events: count=123, start-seq=32, end-seq=154
2018-07-23T22:53:23.946+0800    DEBUG   [memqueue]      memqueue/ackloop.go:111 ackloop: return ack to broker loop:123
2018-07-23T22:53:23.946+0800    DEBUG   [memqueue]      memqueue/ackloop.go:114 ackloop:  done send ack

Hi - could you please help? No clue what to do next. Thanks

Great! And which metricset of system produces this error?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.