Error with HAProxy Dashboards [ 7.10.2 ]

[rodrigo.gz (rodrigo gil)]

Hello!

I am trying to monitor a HAProxy with metricbeat and filebeat. ( 7.10.2 )
When I do the setup and start the agent , it starts collecting metrics and logs correctly but the dashboards do not show the collected information as expected. They show a pike with the correct data and then immediately stop showing real data.

This happens with most visualizations of the default 7 dashboards ( filebeat - metricbeat ).

I have tried visualizing this data via Lens and apparently they are correct. Nevertheless, I would like to be able to have the default dashboards.

I attach a couple of examples.

Thanks in advance!

[Mario_Castro]

Hi @rodrigo.gz

Please, can you also share your configuration file and the log output your are getting from Metricbeat and Filebeat?

[rodrigo.gz (rodrigo gil)]

hello mario!

Thanks for responding so quickly!

This are the 2 metricbeat module haproxy.yml files:

- module: haproxy
  metricsets:
    #- "info"
    - "stat"
  enabled: true
  period: 10s
  hosts: ["http://10.28.28.195:21000/stats","http://10.28.27.42:21000/stats","http://10.28.26.121:21000/stats","http://10.28.31.217:21000/stats","http://10.28.31.156:21000/stats","http://10.28.31.160:21000/stats","http://10.28.29.145:21000/stats","http://10.28.27.175:21000/stats","http://10.28.31.157:21000/stats","http://10.28.31.161:21000/stats","http://10.28.31.141:21000/stats","http://10.28.27.174:21000/stats","http://10.28.30.163:21000/stats","http://10.28.28.178:21000/stats","http://10.28.28.219:21000/stats","http://10.28.28.228:21000/stats","http://10.28.28.85:21000/stats","http://10.28.24.32:21000/stats","http://10.28.31.213:21000/stats","http://10.28.28.245:21000/stats","http://10.28.28.66:21000/stats","http://10.28.30.66:21000/stats","http://10.28.31.224:21000/stats","http://10.28.27.53:21000/stats","http://10.28.30.145:21000/stats","http://10.28.27.173:21000/stats","http://10.28.28.244:21000/stats","http://10.28.31.233:21000/stats","http://10.28.27.18:21000/stats","http://10.28.26.21:21000/stats","http://10.28.29.148:21000/stats","http://10.28.24.16:21000/stats","http://10.28.30.9:21000/stats","http://10.28.31.206:21000/stats","http://10.28.28.76:21000/stats","http://10.28.28.54:21000/stats","http://10.28.31.155:21000/stats","http://10.28.31.159:21000/stats","http://10.28.27.117:21000/stats","http://10.28.31.158:21000/stats","http://10.28.31.153:21000/stats","http://10.28.28.119:21000/stats","http://10.28.31.149:21000/stats","http://10.28.24.14:21000/stats","http://10.28.31.184:21000/stats","http://10.28.30.105:21000/stats","http://10.28.25.95:21000/stats","http://10.28.31.152:21000/stats","http://10.28.28.24:21000/stats","http://10.28.31.225:21000/stats","http://10.28.31.220:21000/stats","http://10.28.26.133:21000/stats","http://10.28.28.208:21000/stats","http://10.28.26.112:21000/stats","http://10.28.28.207:21000/stats","http://10.28.28.209:21000/stats","http://10.28.26.150:21000/stats","http://10.28.28.241:21000/stats","http://10.28.27.235:21000/stats","http://10.28.29.137:21000/stats","http://10.28.31.174:21000/stats","http://10.28.29.133:21000/stats","http://10.28.31.175:21000/stats","http://10.28.31.176:21000/stats","http://10.28.29.134:21000/stats","http://10.28.24.72:21000/stats","http://10.28.29.135:21000/stats","http://10.28.24.74:21000/stats","http://10.28.31.215:21000/stats","http://10.28.28.252:21000/stats","http://10.28.31.234:21000/stats"]

- module: haproxy
  metricsets:
    #- "info"
    - "stat"
  enabled: true
  period: 10s
  hosts: ["http://10.28.31.213:21000/stats","http://10.28.28.76:21000/stats"]
~

And this is the haproxy.yml of the filebeat module:

# Module: haproxy
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.10/filebeat-module-haproxy.html

- module: haproxy
  # All logs
  log:
    enabled: true

    # Set which input to use between syslog (default) or file.
    var.input: "file"

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ['/var/log/haproxy/*/*.log']
~
~

In the logs, metricbeat does not show ERRORS until after 20 minutes it shows a trace like the following:

2021-06-15T11:13:10.042+0200    INFO    module/wrapper.go:259   Error fetching data for metricset haproxy.stat: failed fetching haproxy stat: error making http request: Get "http://10.28.27.175:21000/stats;csv": dial tcp 10.28.27.175:21000: socket: too many open files
2021-06-15T11:13:10.042+0200    INFO    module/wrapper.go:259   Error fetching data for metricset haproxy.stat: failed fetching haproxy stat: error making http request: Get "http://10.28.28.252:21000/stats;csv": dial tcp 10.28.28.252:21000: socket: too many open files
2021-06-15T11:13:10.043+0200    INFO    module/wrapper.go:259   Error fetching data for metricset haproxy.stat: failed fetching haproxy stat: error making http request: Get "http://10.28.27.42:21000/stats;csv": dial tcp 10.28.27.42:21000: socket: too many open files

With this error I have tried to run metricbeat with a higher ulimit sudo sh -c " ulimit -n 65535 && metricbeat -e "
and it has stopped throwing that error and only shows INFO logs with success codes but the dashboards still do not appear correctly.