Hi
Below are my filebeat.yml and filebeat logs, I could not see logs output to Kafka topics (3 topics configured here), and also filebeat registry file didn't change. Could you help to check what went wrong? Also in the page https://www.elastic.co/guide/en/beats/filebeat/master/kafka-output.html, it said "This output works with Kafka 0.8, 0.9, and 0.10", does it mean the Kafka version I have (kafka_2.11-0.10.2.0) wont' work as filebeat output? Thank you!
filebeat.yml:
filebeat:
prospectors:
-
paths:
- "/data/Log/mongodb.log"
input_type: log
close_timeout: 5m
fields:
log_topic: "mongodb_log_filebeat"
-
paths:
- "/var/log/auth.log"
input_type: log
close_timeout: 5m
exclude_lines: ['[.+CRON]* CRON[[0-9]*]: pam_unix((cron:session))']
fields:
log_topic: "Linux_auth_log-Topic"
fields_under_root: true
-
paths:
- "/var/log/syslog"
input_type: log
close_timeout: 5m
include_lines: ['.*EMERG','.*ALERT','.*CRITICAL','.*ERROR','.*WARNING']
fields:
log_topic: "Linux_sys_log-Topic"
fields_under_root: true
output.kafka:
hosts: [":29001", ":29001", ":29001", ":29001", ":29001"]
topic: '%{[fields.log_topic]}'
partition.round_robin:
reachable_only: false
required_acks: 1
Filebeat log:
018-03-19T23:28:14Z INFO Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2018-03-19T23:28:14Z INFO Setup Beat: filebeat; Version: 5.2.0
2018-03-19T23:28:14Z INFO Activated kafka as output plugin.
2018-03-19T23:28:14Z INFO Publisher name:
2018-03-19T23:28:14Z INFO Flush Interval set to: 1s
2018-03-19T23:28:14Z INFO Max Bulk Size set to: 2048
2018-03-19T23:28:14Z INFO filebeat start running.
2018-03-19T23:28:14Z INFO Registry file set to: /var/lib/filebeat/registry
2018-03-19T23:28:14Z INFO Loading registrar data from /var/lib/filebeat/registry
2018-03-19T23:28:14Z INFO States Loaded from registrar: 43
2018-03-19T23:28:14Z INFO Loading Prospectors: 3
2018-03-19T23:28:14Z INFO Starting Registrar
2018-03-19T23:28:14Z INFO Prospector with previous states loaded: 0
2018-03-19T23:28:14Z INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2018-03-19T23:28:14Z INFO Start sending events to output
2018-03-19T23:28:14Z INFO Prospector with previous states loaded: 6
2018-03-19T23:28:14Z INFO Prospector with previous states loaded: 36
2018-03-19T23:28:14Z INFO Loading Prospectors completed. Number of prospectors: 3
2018-03-19T23:28:14Z INFO All prospectors are initialised and running with 43 states to persist
2018-03-19T23:28:14Z INFO Starting prospector of type: log
2018-03-19T23:28:14Z INFO Starting prospector of type: log
2018-03-19T23:28:14Z INFO Harvester started for file: /data/Log/mongodb.log
2018-03-19T23:28:14Z INFO Harvester started for file: /var/log/auth.log
2018-03-19T23:28:14Z INFO Starting prospector of type: log
2018-03-19T23:28:14Z INFO Harvester started for file: /var/log/syslog
2018-03-19T23:28:14Z WARN kafka message: Initializing new client
2018-03-19T23:28:14Z WARN client/metadata fetching metadata for all topics from broker :29001
2018-03-19T23:28:14Z WARN Connected to broker at :29001 (unregistered)
2018-03-19T23:28:14Z WARN client/brokers registered new broker #2 at :29001
2018-03-19T23:28:14Z WARN client/brokers registered new broker #5 at :29001
2018-03-19T23:28:14Z WARN client/brokers registered new broker #4 at :29001
2018-03-19T23:28:14Z WARN client/brokers registered new broker #1 at :29001
2018-03-19T23:28:14Z WARN client/brokers registered new broker #3 at :29001
2018-03-19T23:28:14Z WARN kafka message: Successfully initialized new client
2018-03-19T23:28:14Z WARN producer/broker/1 starting up
2018-03-19T23:28:14Z WARN producer/broker/1 state change to [open] on mongodb_log_filebeat/0
2018-03-19T23:28:14Z WARN producer/broker/1 state change to [open] on mongodb_log_filebeat/2
2018-03-19T23:28:14Z WARN producer/broker/2 starting up
2018-03-19T23:28:14Z WARN producer/broker/3 starting up
2018-03-19T23:28:14Z WARN producer/broker/2 state change to [open] on mongodb_log_filebeat/1
2018-03-19T23:28:14Z WARN producer/broker/3 state change to [open] on mongodb_log_filebeat/4
2018-03-19T23:28:14Z WARN producer/broker/2 state change to [open] on mongodb_log_filebeat/3
2018-03-19T23:28:14Z WARN Connected to broker at :29001 (registered as #1)
2018-03-19T23:28:14Z WARN Connected to broker at :29001 (registered as #3)
2018-03-19T23:28:14Z WARN Connected to broker at :29001 (registered as #2)
2018-03-19T23:28:44Z INFO Non-zero metrics in the last 30s: libbeat.publisher.published_events=1564 libbeat.kafka.published_and_acked_events=1320 filebeat.harvester.open_files=3 libbeat.kafka.call_count.PublishEvents=3165 libbeat.kafka.published_but_not_acked_events=772016 filebeat.harvester.started=3 filebeat.harvester.running=3
2018-03-19T23:29:14Z INFO Non-zero metrics in the last 30s: libbeat.kafka.published_but_not_acked_events=819596 libbeat.kafka.call_count.PublishEvents=3359
...
2018-03-19T23:33:14Z INFO Closing harvester because close_timeout was reached: /var/log/auth.log
2018-03-19T23:33:14Z INFO Closing harvester because close_timeout was reached: /data/Log/mongodb.log
2018-03-19T23:33:14Z INFO Closing harvester because close_timeout was reached: /var/log/syslog
2018-03-19T23:33:44Z INFO Non-zero metrics in the last 30s: libbeat.kafka.published_but_not_acked_events=781776 libbeat.kafka.call_count.PublishEvents=3204
2018-03-19T23:34:14Z INFO Non-zero metrics in the last 30s: libbeat.kafka.call_count.PublishEvents=3147 libbeat.kafka.published_but_not_acked_events=767868
...
2018-03-19T23:38:14Z WARN client/metadata fetching metadata for all topics from broker :29001
2018-03-19T23:38:14Z WARN kafka message: client/metadata got error from broker while fetching metadata:%!(EXTRA *errors.errorString=EOF)
2018-03-19T23:38:14Z WARN Closed connection to broker :29001
2018-03-19T23:38:14Z WARN client/metadata fetching metadata for all topics from broker :29001
2018-03-19T23:38:14Z WARN Connected to broker at :29001 (unregistered)
2018-03-19T23:38:44Z INFO Non-zero metrics in the last 30s: libbeat.kafka.call_count.PublishEvents=3100 libbeat.kafka.published_but_not_acked_events=756400
2018-03-19T23:39:14Z INFO Non-zero metrics in the last 30s: libbeat.kafka.published_but_not_acked_events=799832 libbeat.kafka.call_count.PublishEvents=3278
2018-03-19T23:39:44Z INFO Non-zero metrics in the last 30s: libbeat.kafka.published_but_not_acked_events=777628 libbeat.kafka.call_count.PublishEvents=3187
...
2018-03-19T23:48:14Z WARN client/metadata fetching metadata for all topics from broker :29001