Configuration example of filebeat haproxy module

I'm trying to get the filebeat haproxy module to work. I probably need one working example (or a list of what is required from all the dependencies)

haproxy version
1.7.7

filebeat -version
Flag --version has been deprecated, use version subcommand
filebeat version 6.5.4 (amd64), libbeat 6.5.4

/opt/elasticsearch/bin/elasticsearch -version
Version: 6.5.4, Build: oss/tar/d2ef93d/2018-12-17T21:17:40.758843Z, JVM: 1.8.0_191

Current configuration

haproxy.config

global
   log 127.0.0.1:9001 local0 debug
   log 127.0.0.1:9001 local1 debug

defaults
    log global
    mode http
    option httplog 
    option dontlognull

frontend test
  mode http
  option httplog
  log global
  default_backend b_test

backend b_test
  ..

filebeat configuration

filebeat.modules:
  - module: nginx
    access:
      enabled: true #
    error:
      enabled: true

  - module: haproxy
    log:
      enabled: true

The filebeat shows the message is send

2019-01-15T20:31:18.336Z        DEBUG   [publish]       pipeline/processor.go:308       Publish event: {                                  
  "@timestamp": "2019-01-15T20:31:18.000Z",                                                                                                         
  "@metadata": {                                                                                                                      
    "beat": "filebeat",                                                                                                             
    "type": "doc",                                                                                                                                                  
    "version": "6.5.4",                                                                                       
    "truncated": false,                                                                                                                   
    "pipeline": "filebeat-6.5.4-haproxy-log-pipeline"                                                                      
  },                                                                                                                                      
  "hostname": "haproxy[7594]:",                                                                                                                                      
  "message": "127.0.0.1:40042 [15/Jan/2019:20:31:18.331] lrs b_lrs/frontend-phoeix 0/0/0/1/1 200 284 - - ---- 0/0/0/0/0 0/0 \"GET /test HTTP/1.0\"",
  "fileset": {                                                                                                                        
    "module": "haproxy",                                                                                                        
    "name": "log"                                                                                          
  },                                                                                                                               
  "prospector": {                                                                                                                      
    "type": "syslog"                                                                                                                                 
  },                                                                                                                                
  "host": {                                                                                                                         
    "name": "frontend-phoeix"                                                                                                        
  },                                                                                                                                
  "source": "127.0.0.1:44693",                                                                                                                
  "event": {                                                                                                                                        
    "severity": 6                                                                                                          
  },                                                                                                                                  
  "process": {                                                                                                                  
    "program": "127.0.0.1"                                                                                                                                           
  },                                                                                                                       
  "input": {                                                                                                                                         
    "type": "syslog"                                                                                                                   
  },                                                                                                                                                 
  "beat": {                                                                                                                               
    "name": "frontend-phoeix",                                                                                                                      
    "hostname": "frontend-phoeix",                                                                                                    
    "version": "6.5.4"                                                                                                              
  },                                                                                                                                
  "syslog": {                                                                                                    
    "priority": 142,                                                                                 
    "severity_label": "Informational",                                                                                     
    "facility": 17,                                                                                                         
    "facility_label": "local1"                                                                                                                      
  }                                                                                                                                   
} 

But the haproxy dashboard is looking for
haproxy.backend_name

what am I missing?

I actually did found an error. So I'm choking on an Grok expression.

Provided Grok expressions do not match field value: [127.0.0.1:40042 [15/Jan/2019:20:31:18.331] lrs b_lrs/frontend-phoeix 0/0/0/1/1 200 284 - - ---- 0/0/0/0/0 0/0 \"GET /test HTTP/1.0\"]

I got it finally working after putting additional capture flag in haproxy.cfg

What leaves my config now at

global
    log 127.0.0.1:9001 local0
    log 127.0.0.1:9001 local1 debug
    user haproxy
    group haproxy
    daemon
    ssl-server-verify none

defaults
    log     global
    mode    http
    option  httplog
    timeout connect 5000
    timeout client  50000
    timeout server  50000
    timeout http-keep-alive 1s

frontend lrs
    bind *:10100
    capture request header Host len 15
    capture request header X-Forwarded-For len 15
    capture request header Referer len 15
    capture response header Content-length len 9
    capture response header Location len 15
    default_backend b_lrs

backend b_lrs
    balance roundrobin
    capture request header Host len 15
    capture request header X-Forwarded-For len 15
    capture request header Referer len 15
    capture response header Content-length len 9
    capture response header Location len 15
    server frontend-phoeix 172.19.0.2:8070 check

But it took me another few hours to get the dashboard to work

I think the main issue was that I have old beats posting version 6.3.2 into a 6.5 elastic search.

So basically you get multiple different document versions going into ES.

Is there a way to stop this on the server side?

E.g. only access beats 6.5.* and reject any other document inserts?

The out of the box experience on updating beats, or enabling new modules
has not been ideal. I understand there are a lot of moving parts. Would be great if we can put some restrictions and get going more easily

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.