Filebeat container is starting listening UDP need to change to TCP

docker

(Ranjith M) #1

Filebeat Version : 6.5

We need to know how can configure the haproxy module for filebeat . Currently the filebeat container is starting listening UDP connections in localhost:9001, we need to know how change the protocol and the address throuth module configuration. The desired configuration should be {"protocol": "tcp"} and connection {"address": ":9001"}.

Below a trace for container start.

2018-11-15T16:12:47.540Z DEBUG [cfgfile] cfgfile/reload.go:176 Scan for new config files
2018-11-15T16:12:47.541Z DEBUG [cfgfile] cfgfile/cfgfile.go:177 Load config from file: /usr/share/filebeat/modules.d/haproxy.yml
2018-11-15T16:12:47.541Z DEBUG [cfgfile] cfgfile/reload.go:195 Number of module configs found: 1
2018-11-15T16:12:47.541Z DEBUG [reload] cfgfile/list.go:62 Starting reload procedure, current runners: 0
2018-11-15T16:12:47.541Z DEBUG [reload] cfgfile/list.go:80 Start list: 1, Stop list: 0
2018-11-15T16:12:47.541Z WARN [cfgwarn] syslog/input.go:111 EXPERIMENTAL: Syslog input type is used
2018-11-15T16:12:47.541Z DEBUG [processors] processors/processor.go:66 Processors:
2018-11-15T16:12:47.541Z DEBUG [reload] cfgfile/list.go:101 Starting runner: haproxy (log)
2018-11-15T16:12:47.541Z INFO input/input.go:114 Starting input of type: syslog; ID: 14175615836827865463
2018-11-15T16:12:47.541Z INFO cfgfile/reload.go:205 Loading of config files completed.
2018-11-15T16:12:47.541Z INFO [syslog] syslog/input.go:173 Starting Syslog input {"protocol": "udp"}
2018-11-15T16:12:47.545Z INFO [udp] udp/server.go:66 Started listening for UDP connection {"address": "localhost:9001"}

Configuration File

filebeat.yml:
filebeat.config:
inputs:
path: {path.config}/inputs.d/*.yml reload.enabled: false modules: path: {path.config}/modules.d/*.yml
reload.enabled: false

output.kafka:
  hosts: ["XXXXXXXXXXXXXXXXXXXXXXXX"]
  topic: 'XXXXXXXXXXX'
  partition.round_robin:
    reachable_only: false

  required_acks: 1
  compression: gzip
  max_message_bytes: 1000000
  version: '0.10.2'

logging.level: debug

  haproxy.yml: |-
    - module: haproxy
      log:
        enabled: true

(Steffen Siering) #2

You can try to use the module input configuration to disable the udp input and enable tcp:

- module: haproxy
  log.enabled: true
  input:
    protocol.udp.enabled: false
    protocol.tcp.host: "localhost:9001"

(Jose Tomas) #3

With this configuration :

2018-11-19T12:31:31.412Z ERROR instance/beat.go:824 Exiting: Fileset haproxy/input is configured but doesn't exist


(Steffen Siering) #4

There is an error in the sample config. This should work:

- module: haproxy
  log:
    enabled: true
    input:
      protocol.udp.enabled: false
      protocol.tcp.host: "localhost:9001"

(Jose Tomas) #5

Thanks for you support, currently with the last format conf is working but :

  • We have set configure "option httplog" in haproxy's side but the events are being sent without any format. In metadata struct in pipeline field you can see filebeat-6.5.0-haproxy-log-pipeline, but no any format comes. The fileset is empty, only in message field have the access log.
  • Have filebeat haproxy's module any requirement about Haproxy's version?

"@timestamp": "2018-11-20T10:47:06.000Z",
"@metadata": {
"beat": "filebeat",
"type": "doc",
"version": "6.5.0",
"truncated": false,
"pipeline": "filebeat-6.5.0-haproxy-log-pipeline"
},
"source": "127.0.0.1:43424",
"process": {},
"fileset": {
"name": "log",
"module": "haproxy"
},
"prospector": {
"type": "syslog"
},
"input": {
"type": "syslog"
},
"beat": {
"name": "XXXXXXXXXXXXXXXXXX",
"hostname": "XXXXXXXXXXXXXXXX",
"version": "6.5.0"
},
"host": {
"name": "XXXXXXXXXXXXXXXXXX"
},
"message": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
"syslog": {
"priority": 142,
"severity_label": "Informational",
"facility": 17,
"facility_label": "local1"
},
"event": {
"severity": 6
}
}