Filebeat not publishing logs to KAFKA

Elastic Stack Beats
1

Filebeat version: Filebeat-5.0.0-beta1

run with command : ./filebeat -c -e filebeat.yml -d "*"

Output logs

2016/09/28 18:03:11.714193 output.go:109: DBG output worker: publish 588 events
2016/09/28 18:03:11.714230 lb.go:72: DBG configure maxattempts: 0
2016/09/28 18:03:11.714292 async_worker.go:66: DBG load balancer: start client loop
2016/09/28 18:03:11.714322 client.go:61: DBG connect: [x.x.x.x:9092]
2016/09/28 18:03:11.714340 log.go:16: WARN kafka message: Initializing new client
2016/09/28 18:03:11.714402 log.go:12: WARN client/metadata fetching metadata for all topics from broker 10.96.72.26:9092
2016/09/28 18:03:11.714455 context.go:93: DBG forwards msg with attempts=-1
2016/09/28 18:03:11.714924 log.go:12: WARN Connected to broker at x.x.x.x:9092 (unregistered)
2016/09/28 18:03:11.716125 log.go:12: WARN client/brokers registered new broker #2 at hostname-01:9092
2016/09/28 18:03:11.716138 log.go:12: WARN client/brokers registered new broker #3 at hostname-01:9092
2016/09/28 18:03:11.716147 log.go:12: WARN client/brokers registered new broker #0 at hostname-01:9092
2016/09/28 18:03:11.716233 log.go:16: WARN kafka message: Successfully initialized new client
2016/09/28 18:03:11.716265 context.go:138: DBG events from worker worker queue
2016/09/28 18:03:11.716281 client.go:104: DBG publish events
2016/09/28 18:03:11.716359 context.go:98: DBG message forwarded
2016/09/28 18:03:11.716587 log.go:12: WARN producer/broker/3 starting up
2016/09/28 18:03:11.716602 log.go:12: WARN producer/broker/3 state change to [open] on beats/0
2016/09/28 18:03:11.718304 log.go:12: WARN Connected to broker at hostname-01:9092 (registered as #3)
2016/09/28 18:03:11.736660 client.go:210: DBG finished kafka batch
2016/09/28 18:03:11.736686 client.go:215: DBG Kafka publish failed with: kafka server: Unexpected (unknown?) server error.
2016/09/28 18:03:11.736698 async_worker.go:152: DBG handleResults
2016/09/28 18:03:11.736709 async_worker.go:155: DBG handle publish error: kafka server: Unexpected (unknown?) server error.
2016/09/28 18:03:11.736722 context.go:136: DBG events from retries queue
2016/09/28 18:03:11.736734 client.go:104: DBG publish events
2016/09/28 18:03:11.747406 client.go:210: DBG finished kafka batch
2016/09/28 18:03:11.747445 client.go:215: DBG Kafka publish failed with: kafka server: Unexpected (unknown?) server error.
2016/09/28 18:03:11.747453 async_worker.go:152: DBG handleResults
2016/09/28 18:03:11.747460 async_worker.go:155: DBG handle publish error: kafka server: Unexpected (unknown?) server error.

Questions:

  1. When it successfully connects to KAFKA broker , should not it be INFO message than WARN message?
  2. Why does it say Connected to broker at x.x.x.x:9092 (unregistered). What does unregistered mean here?
  3. kafka server: Unexpected (unknown?) server error - does not say much about what happened.
  4. Anyone has got Kafka output working with Filebeat-5.0.0-beta1? How does your config looks like?

Thank you in advance

2

Below is the output from KAFKA

[2016-09-28 14:56:10,719] ERROR [Replica Manager on Broker 3]: Error processing append operation on partition beats-0 (kafka.server.ReplicaManager)
java.lang.IllegalStateException: Compressed message has magic value 0 but inner message has magic value 1

3

Its the version: 0.10.0.0 in filebeat.yml which causes the issue.

If I comment this line I am able to see the logs in Kafka topic.

Will Close this issue.

1 Like
4

Thanks for reporting. Can you open an issue on github? I'd like to mark it for the elastic pioneer program.

5

Done.

6

This topic was automatically closed after 21 days. New replies are no longer allowed.