I am having an issue with the kafka module where it does not appear to process the lines properly. I have enabled debugging of filebeat and see the following:
2018-03-16T18:36:54Z DBG [publish] Publish event: { "@timestamp": "2018-03-16T18:36:54.052Z", "@metadata": { "beat": "filebeat", "type": "doc", "version": "6.1.2", "pipeline": "filebeat-6.1.2-kafka-log-pipeline" }, "source": "/var/log/kafka/controller.log", "offset": 84336, "message": "[2018-03-16 18:36:45,144] DEBUG [Controller 2]: topics not in preferred replica Map() (kafka.controller.KafkaController)", "fileset": { "name": "log", "module": "kafka" }, "prospector": { "type": "log" }, "beat": { "name": "eventing2030-ct-kafka-2", "hostname": "eventing2030-ct-kafka-2", "version": "6.1.2" } }
The line appears to match what the module shows so I'm not sure why its not being processed. The syslog module on the other hand is working fine.
I am referring to this https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-module-kafka.html
Based on this document I expect filebeats to read the log lines, parse it and create appropriate fields before sending it to kafka(our logging buffer; not the same as the kafka where the filebeats is running)
Any assistance would be appreciated.
Thanks,
Ryan