# Filebeat not publishing logs to KAFKA **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [September 28, 2016, 6:20pm UTC](https://discuss.elastic.co/t/filebeat-not-publishing-logs-to-kafka/61733 "2016-09-28T18:20:16Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Tarini\_Dash](https://avatars.discourse-cdn.com/v4/letter/t/76d3ee/32.png) [@Tarini\_Dash](https://discuss.elastic.co/u/Tarini_Dash)\ **Post date:** [September 28, 2016, 6:20pm UTC](https://discuss.elastic.co/t/filebeat-not-publishing-logs-to-kafka/61733/1 "2016-09-28T18:20:17Z")
Filebeat version: Filebeat-5.0.0-beta1 run with command : ./filebeat -c -e filebeat.yml -d "\*" # Output logs 2016/09/28 18:03:11.714193 output.go:109: DBG output worker: publish 588 events 2016/09/28 18:03:11.714230 lb.go:72: DBG configure maxattempts: 0 2016/09/28 18:03:11.714292 async\_worker.go:66: DBG load balancer: start client loop 2016/09/28 18:03:11.714322 client.go:61: DBG connect: [x.x.x.x:9092] 2016/09/28 18:03:11.714340 log.go:16: WARN kafka message: Initializing new client 2016/09/28 18:03:11.714402 log.go:12: WARN client/metadata fetching metadata for all topics from broker 10.96.72.26:9092 2016/09/28 18:03:11.714455 context.go:93: DBG forwards msg with attempts=-1 2016/09/28 18:03:11.714924 log.go:12: WARN Connected to broker at x.x.x.x:9092 (unregistered) 2016/09/28 18:03:11.716125 log.go:12: WARN client/brokers registered new broker #2 at hostname-01:9092 2016/09/28 18:03:11.716138 log.go:12: WARN client/brokers registered new broker #3 at hostname-01:9092 2016/09/28 18:03:11.716147 log.go:12: WARN client/brokers registered new broker #0 at hostname-01:9092 2016/09/28 18:03:11.716233 log.go:16: WARN kafka message: Successfully initialized new client 2016/09/28 18:03:11.716265 context.go:138: DBG events from worker worker queue 2016/09/28 18:03:11.716281 client.go:104: DBG publish events 2016/09/28 18:03:11.716359 context.go:98: DBG message forwarded 2016/09/28 18:03:11.716587 log.go:12: WARN producer/broker/3 starting up 2016/09/28 18:03:11.716602 log.go:12: WARN producer/broker/3 state change to [open] on beats/0 2016/09/28 18:03:11.718304 log.go:12: WARN Connected to broker at hostname-01:9092 (registered as #3) 2016/09/28 18:03:11.736660 client.go:210: DBG finished kafka batch 2016/09/28 18:03:11.736686 client.go:215: DBG Kafka publish failed with: kafka server: Unexpected (unknown?) server error. 2016/09/28 18:03:11.736698 async\_worker.go:152: DBG handleResults 2016/09/28 18:03:11.736709 async\_worker.go:155: DBG handle publish error: kafka server: Unexpected (unknown?) server error. 2016/09/28 18:03:11.736722 context.go:136: DBG events from retries queue 2016/09/28 18:03:11.736734 client.go:104: DBG publish events 2016/09/28 18:03:11.747406 client.go:210: DBG finished kafka batch 2016/09/28 18:03:11.747445 client.go:215: DBG Kafka publish failed with: kafka server: Unexpected (unknown?) server error. 2016/09/28 18:03:11.747453 async\_worker.go:152: DBG handleResults 2016/09/28 18:03:11.747460 async\_worker.go:155: DBG handle publish error: kafka server: Unexpected (unknown?) server error. Questions: 1. When it successfully connects to KAFKA broker , should not it be INFO message than WARN message? 2. Why does it say _Connected to broker at x.x.x.x:9092 ( **unregistered** )_. What does unregistered mean here? 3. _kafka server: Unexpected (unknown?) server error_ - does not say much about what happened. 4. Anyone has got Kafka output working with Filebeat-5.0.0-beta1? How does your config looks like? Thank you in advance ---
**Author:** ![Tarini\_Dash](https://avatars.discourse-cdn.com/v4/letter/t/76d3ee/32.png) [@Tarini\_Dash](https://discuss.elastic.co/u/Tarini_Dash)\ **Post date:** [September 28, 2016, 6:57pm UTC](https://discuss.elastic.co/t/filebeat-not-publishing-logs-to-kafka/61733/2 "2016-09-28T18:57:36Z")
# Below is the output from KAFKA [2016-09-28 14:56:10,719] ERROR [Replica Manager on Broker 3]: Error processing append operation on partition beats-0 (kafka.server.ReplicaManager) java.lang.IllegalStateException: Compressed message has magic value 0 but inner message has magic value 1 ---
**Author:** ![Tarini\_Dash](https://avatars.discourse-cdn.com/v4/letter/t/76d3ee/32.png) [@Tarini\_Dash](https://discuss.elastic.co/u/Tarini_Dash)\ **Post date:** [September 28, 2016, 7:21pm UTC](https://discuss.elastic.co/t/filebeat-not-publishing-logs-to-kafka/61733/3 "2016-09-28T19:21:41Z")
Its the **version: 0.10.0.0** in filebeat.yml which causes the issue. If I comment this line I am able to see the logs in Kafka topic. Will Close this issue. ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [September 29, 2016, 10:04am UTC](https://discuss.elastic.co/t/filebeat-not-publishing-logs-to-kafka/61733/4 "2016-09-29T10:04:55Z")
Thanks for reporting. Can you open an issue on [github](https://github.com/elastic/beats/issues)? I'd like to mark it for the [elastic pioneer program](https://www.elastic.co/blog/elastic-pioneer-program). ---
**Author:** ![Tarini\_Dash](https://avatars.discourse-cdn.com/v4/letter/t/76d3ee/32.png) [@Tarini\_Dash](https://discuss.elastic.co/u/Tarini_Dash)\ **Post date:** [September 29, 2016, 1:52pm UTC](https://discuss.elastic.co/t/filebeat-not-publishing-logs-to-kafka/61733/5 "2016-09-29T13:52:17Z")
Done. > ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 19, 2016, 6:20pm UTC](https://discuss.elastic.co/t/filebeat-not-publishing-logs-to-kafka/61733/6 "2016-10-19T18:20:20Z")
This topic was automatically closed after 21 days. New replies are no longer allowed.