Good morning,
I have one machine with filebeat that send data to kafka. The problem is that not send data a kafka and it not register never errors in the log of filebeat.
there is telnet connectivity and I did the test of sending data with kafka to the topic and it works correctly.
I attach the configurations and versions of the products:
Ubuntu 20.04
Filebeat version 7.14.1
Kafka version 2.8.0
Configuration filebeat:
filebeat.inputs:
- type: syslog
enabled: false
paths:
- /var/log/*.log
- /var/log/syslog
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 1
output.kafka:
enabled: true
hosts: ["127.0.0.1:9092"]
topic: "graylog-events"
codec.json:
pretty: false
logging.level: debug
logging.selectors: ["*"]
Log filebeat:
Sep 13 11:03:24 core-graylog-carmelo filebeat[3557]: 2021-09-13T11:03:24.090+0200#011INFO#011[monitoring]#011log/log.go:145#011Non-zero metrics in the last 30s#011{"monitoring": {"metrics": {"beat":{"cgroup":{"cpuacct":{"total":{"ns":6766681}},"memory":{"mem":{"usage":{"bytes":-126976}}}},"cpu":{"system":{"ticks":160,"time":{"ms":7}},"total":{"ticks":560,"time":{"ms":12},"value":560},"user":{"ticks":400,"time":{"ms":5}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":10},"info":{"ephemeral_id":"69e76a2c-80ff-4815-ab2b-4244fc9765dc","uptime":{"ms":600124},"version":"7.14.1"},"memstats":{"gc_next":18602640,"memory_alloc":11827760,"memory_total":68840608,"rss":106889216},"runtime":{"goroutines":23}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":1}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.03,"15":0.24,"5":0.14,"norm":{"1":0.015,"15":0.12,"5":0.07}}}}}}
Sep 13 11:03:24 core-graylog-carmelo filebeat[3557]: 2021-09-13T11:03:24.171+0200#011DEBUG#011[input]#011input/input.go:139#011Run input
Sep 13 11:03:24 core-graylog-carmelo filebeat[3557]: 2021-09-13T11:03:24.171+0200#011DEBUG#011[input]#011log/input.go:215#011Start next scan#011{"input_id": "4f935c25-36fd-4121-94a3-4f47e0eddad4"}
Sep 13 11:03:24 core-graylog-carmelo filebeat[3557]: 2021-09-13T11:03:24.171+0200#011DEBUG#011[input]#011log/input.go:279#011input states cleaned up. Before: 0, After: 0, Pending: 0#011{"input_id": "4f935c25-36fd-4121-94a3-4f47e0eddad4"}