Module haproxy : error getting IdlePct: strconv.ParseFloat: parsing \"\": invalid syntax

Elastic Stack Beats
1
  • Version: metricbeat 7.4.0 haproxy 1.9.8
  • Operating System: Ubuntu 16.04.6 LTS
  • Discuss Forum URL: Haproxy strconv.ParseFloat: parsing "": invalid syntax
  • Steps to Reproduce: install mesos/marthon loadbalancer (which contains the haproxy) , install metricbeat with haproxy module (7.4.0)

got the same error, found no solution, is this a bug? :

Okt 08 12:00:53 ba-load-001 metricbeat[4693]: 2019-10-08T12:00:53.191+0200 INFO module/wrapper.go:252 Error fetching data for metricset haproxy.info: error in mapping: error getting IdlePct: strconv.ParseFloat: parsing "": invalid syntax
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: 2019-10-08T12:00:53.191+0200 DEBUG [processors] processing/processors.go:183 Publish event: {
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: "@timestamp": "2019-10-08T10:00:53.191Z",
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: "@metadata": {
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: "beat": "metricbeat",
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: "type": "_doc",
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: "version": "7.4.0"
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: },
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: "event": {
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: "dataset": "haproxy.info",
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: "module": "haproxy",
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: "duration": 490254
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: },
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: "metricset": {
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: "name": "info",
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: "period": 10000
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: },
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: "service": {
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: "type": "haproxy",
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: "address": "ba-load-001:9090"
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: },
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: "error": {
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: "message": "error in mapping: error getting IdlePct: strconv.ParseFloat: parsing "": invalid syntax"
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: },
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: "host": {
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: "id": "316de4b744b437c1e37d543c5ac4a02f",
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: "containerized": false,
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: "hostname": "ba-load-001",
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: "architecture": "x86_64",
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: "os": {
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: "name": "Ubuntu",
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: "kernel": "4.4.0-154-generic",
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: "codename": "xenial",
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: "platform": "ubuntu",
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: "version": "16.04.6 LTS (Xenial Xerus)",
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: "family": "debian"
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: },
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: "name": "ba-load-001"
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: },
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: "agent": {
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: "type": "metricbeat",
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: "ephemeral_id": "283d7793-68dd-47ee-b9c9-c2a5f35945ed",
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: "hostname": "ba-load-001",
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: "id": "88908c75-57c7-490a-972b-fe1dbf5b4806",
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: "version": "7.4.0"
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: },
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: "ecs": {
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: "version": "1.1.0"
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: }
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: }
Okt 08 12:00:53 ba-load-001 metricbeat[4693]: 2019-10-08T12:00:53.192+0200 DEBUG [processors] processing/processors.go:183 Publish event: {

:

2

@bodo.te

Can you send us your metricset config? If you already know how to send a show info command to haproxy's socket, can you send us that? If not, we can try and figure it out.

3

Is this what you need?:

root@305a298bcc7c:/marathon-lb# echo 'show info' | socat /var/run/haproxy/socket -
Name: HAProxy
Version: 1.9.8
Release_date: 2019/05/13
Nbthread: 1
Nbproc: 1
Process_num: 1
Pid: 8608
Uptime: 0d 0h10m10s
Uptime_sec: 610
Memmax_MB: 0
PoolAlloc_MB: 0
PoolUsed_MB: 0
PoolFailed: 0
Ulimit-n: 100313
Maxsock: 100313
Maxconn: 50000
Hard_maxconn: 50000
CurrConns: 1
CumConns: 681
CumReq: 3035
MaxSslConns: 0
CurrSslConns: 1
CumSslConns: 657
Maxpipes: 0
PipesUsed: 0
PipesFree: 0
ConnRate: 0
ConnRateLimit: 0
MaxConnRate: 27
SessRate: 0
SessRateLimit: 0
MaxSessRate: 27
SslRate: 0
SslRateLimit: 0
MaxSslRate: 27
SslFrontendKeyRate: 0
SslFrontendMaxKeyRate: 28
SslFrontendSessionReuse_pct: 0
SslBackendKeyRate: 0
SslBackendMaxKeyRate: 0
SslCacheLookups: 325
SslCacheMisses: 17
CompressBpsIn: 0
CompressBpsOut: 0
CompressBpsRateLim: 0
ZlibMemUsage: 0
MaxZlibMemUsage: 0
Tasks: 627
Run_queue: 2
Idle_pct: 100
node: 305a298bcc7c
Stopping: 0
Jobs: 184
Unstoppable Jobs: 0
Listeners: 182
ActivePeers: 0
ConnectedPeers: 0
DroppedLogs: 413
BusyPolling: 0

root@305a298bcc7c:/marathon-lb#
4

I can't seem to reproduce this with stock HAProxy 1.9.8. Are these failures sporadic or consistent?

5

@bodo.te Could you share your metricbeat config as well?

6

I have a suspicion as to what's going on here. The error message is unhelpful and unrelated (which is a problem) and the actual issue is some connection problem between metricbeat and haproxy, such that we're getting back an empty object, hence the "" thing.

7

it is always, never got it to work

8

Also : the haproxy runs inside a docker container, (which is , what mesos/marathon requires), but the metricbeat runs on the same host but outside the docker container.
Now that you say it's probably an empty reply , and i lock in to my config , which is:

# Module: haproxy
# Docs: https://www.elastic.co/guide/en/beats/metricbeat/7.4/metricbeat-module-haproxy.html

- module: haproxy
   metricsets:
    - info
    - stat
  period: 10s
  hosts: ["tcp://ba-load-001:9090"]

Now I can see where this problem comes from:
I just forgot to map the port 9090 to the outside of the container.

ON the other hand:
Whereas I have just one haproxy container on my preproduction system.
I have around 10 haproxy containers on my production setup. how would that work with the meticbeat haproxy-module?
should i try to run in inside the docker-container ?

Apart from that : maybe a better error message for empty replys would be helpful?

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.