Can’t get HAPROXY to send stats or info data to Metricbeat

Elastic Stack Beats
,
1 
- module: haproxy
  metricsets: ["info", "stat"]
  period: 10s
  hosts: ["tcp://[unique_host]:14567"]
  username : "haproxy"
  password : "haproxy"
  enabled: true

global
    log         127.0.0.1 local0

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    # STIG'd to V-240092
    maxconn     32768
    user        haproxy
    group       haproxy
    daemon

    stats socket 127.0.0.1:14567 user haproxy group haproxy mode 660 level admin

    # turn on stats unix socket
    #stats socket /var/lib/haproxy/stats
    # STIG'd to V-240058
    stats socket /var/run/haproxy.sock user haproxy group haproxy mode 600 level admin
    

    # utilize system-wide crypto-policies
    ssl-default-bind-ciphers PROFILE=SYSTEM
    ssl-default-server-ciphers PROFILE=SYSTEM

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    tcp
    log                     global
    option                  dontlognull
    stats auth haproxy:haproxy
    retries                 3
    # STIG'd to V-240039
    timeout http-request    5000s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout client-fin      30s
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s

#---------------------------------------------------------------------
# Stats Listener
#---------------------------------------------------------------------

 listen stats
   bind 0.0.0.0:14567
   stats enable
   stats uri /stats
   stats auth haproxy:haproxy

Above are the haproxy yml for metricbeat and the haproxy.cfg.
While metricbeats is running, HAProxy gives these logs:

000000c1:stats.accept(000b)=0012 from [[IP]:35990] ALPN=
000000c1:stats.clicls[adfd:ffffffff]
000000c1:stats.closed[adfd:ffffffff]
000000c2:stats.accept(000b)=0013 from [[IP]:39830] ALPN=
000000c3:stats.accept(000b)=0012 from [[IP]:39828] ALPN=
000000c3:stats.clicls[adfd:ffffffff]
000000c2:stats.clicls[adfd:ffffffff]
000000c3:stats.closed[adfd:ffffffff]
000000c2:stats.closed[adfd:ffffffff]
000000c4:stats.accept(000b)=0012 from [[IP]:39834] ALPN=
000000c5:stats.accept(000b)=0013 from [[IP]:39836] ALPN=
000000c4:stats.clicls[adfd:ffffffff]
000000c5:stats.clicls[adfd:ffffffff]
000000c5:stats.closed[adfd:ffffffff]
000000c4:stats.closed[adfd:ffffffff]

So HAProxy is obviously getting hit by Metricbeat.

But on Metricbeat side I get this:

{“log.level”:“error”,“@timestamp”:“2024-04-08T19:35:59.000Z”,“log.origin”:{“function”:“github.com/elastic/beats/v7/metricbeat/mb/module.(*metricSetWrapper).fetch",“file.name”:“module/wrapper.go”,“file.line”:256},“message”:"Error fetching data for metricset haproxy.info: failed fetching haproxy info: got empty response from HAProxy”,“service.name”:“metricbeat”,“ecs.version”:“1.6.0”}
{“log.level”:“error”,“@timestamp”:“2024-04-08T19:35:59.000Z”,“log.origin”:{“function”:“github.com/elastic/beats/v7/metricbeat/mb/module.(*metricSetWrapper).fetch",“file.name”:“module/wrapper.go”,“file.line”:256},“message”:"Error fetching data for metricset haproxy.stat: failed fetching haproxy stat: got empty response from HAProxy”,“service.name”:“metricbeat”,“ecs.version”:“1.6.0”}
{“log.level”:“error”,“@timestamp”:“2024-04-08T19:36:09.103Z”,“log.origin”:{“function”:“github.com/elastic/beats/v7/metricbeat/mb/module.(*metricSetWrapper).fetch",“file.name”:“module/wrapper.go”,“file.line”:256},“message”:"Error fetching data for metricset haproxy.info: failed fetching haproxy info: got empty response from HAProxy”,“service.name”:“metricbeat”,“ecs.version”:“1.6.0”}
{“log.level”:“error”,“@timestamp”:“2024-04-08T19:36:09.103Z”,“log.origin”:{“function”:“github.com/elastic/beats/v7/metricbeat/mb/module.(*metricSetWrapper).fetch",“file.name”:“module/wrapper.go”,“file.line”:256},“message”:"Error fetching data for metricset haproxy.stat: failed fetching haproxy stat: got empty response from HAProxy”,“service.name”:“metricbeat”,“ecs.version”:“1.6.0”}

When it hits HAProxy for stats and info, there’s an empty response.

Any suggestions?

2

Hello,

Could you try to add /stats to your hosts config?

- module: haproxy
  metricsets: ["info", "stat"]
  period: 10s
  hosts: ["tcp://[unique_host]:14567/stats"]
  username : "haproxy"
  password : "haproxy"
  enabled: true

That would be roughly what is working for us. But we are not using the info metricset so not sure if that will also work with this.

3

Thank you for the response!

I made the change and still get empty responses from the server:

{"log.level":"error","@timestamp":"2024-04-09T11:58:01.322Z","log.origin":{"function":"github.com/elastic/beats/v7/metricbeat/mb/module.(*metricSetWrapper).fetch","file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset haproxy.info: failed fetching haproxy info: got empty response from HAProxy","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-04-09T11:58:01.324Z","log.origin":{"function":"github.com/elastic/beats/v7/metricbeat/mb/module.(*metricSetWrapper).fetch","file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset haproxy.stat: failed fetching haproxy stat: got empty response from HAProxy","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-04-09T11:58:11.325Z","log.origin":{"function":"github.com/elastic/beats/v7/metricbeat/mb/module.(*metricSetWrapper).fetch","file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset haproxy.stat: failed fetching haproxy stat: got empty response from HAProxy","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-04-09T11:58:11.326Z","log.origin":{"function":"github.com/elastic/beats/v7/metricbeat/mb/module.(*metricSetWrapper).fetch","file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset haproxy.info: failed fetching haproxy info: got empty response from HAProxy","service.name":"metricbeat","ecs.version":"1.6.0"}
4

Here's an update:

Using telnet command from the host with haproxy, I can access the server and send it commands.

Using telnet from the host with metricbeat, it connects however is immediately dropped with "Connection closed by foreign host"

5

I also get this in the haproxy logs:

[WARNING] 099/162211 (8) : config : 'stats' statement ignored for proxy 'stats' as it requires HTTP mode.

But we also don't want HTTP - we need TCP especially since info metricset can't be used through http.