Issue with Kafka Module

[ryan.dyer]

I am having an issue with the kafka module not appear to process the lines properly. I have enabled debugging of filebeat and see the following:
2018-03-16T18:36:54Z DBG [publish] Publish event: { "@timestamp": "2018-03-16T18:36:54.052Z", "@metadata": { "beat": "filebeat", "type": "doc", "version": "6.1.2", "pipeline": "filebeat-6.1.2-kafka-log-pipeline" }, "source": "/var/log/kafka/controller.log", "offset": 84336, "message": "[2018-03-16 18:36:45,144] DEBUG [Controller 2]: topics not in preferred replica Map() (kafka.controller.KafkaController)", "fileset": { "name": "log", "module": "kafka" }, "prospector": { "type": "log" }, "beat": { "name": "eventing2030-ct-kafka-2", "hostname": "eventing2030-ct-kafka-2", "version": "6.1.2" } }

The line appears to match what the module shows so I'm not sure why its not being processed. The syslog module on the other hand is working fine.

Any assistance would be appreciated.

Thanks

[ryan.dyer]

Anyone have any ideas on this?

[ruflin]

Can you share what you mean by not processed? The processing of the log lines happens in the Ingest Pipeline in Elasticsearch and not on the Filebeat side. So it is expected that the published event above is not processed yet.