# We are not receiving the logs from filebeat to kafka
**URL:** https://discuss.elastic.co/t/we-are-not-receiving-the-logs-from-filebeat-to-kafka/178756
**Category:** Beats
**Tags:** filebeat
**Created:** [April 27, 2019, 1:53pm UTC](https://discuss.elastic.co/t/we-are-not-receiving-the-logs-from-filebeat-to-kafka/178756 "2019-04-27T13:53:33Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@saravananveera](https://discuss.elastic.co/u/saravananveera)
#### Post date: [April 27, 2019, 1:53pm UTC](https://discuss.elastic.co/t/we-are-not-receiving-the-logs-from-filebeat-to-kafka/178756/1 "2019-04-27T13:53:34Z")
Hi
We are using kafka 2.12 and configure the output as below in the filebeat.
service versions:
1. logstash-6.3.1
2. kafka\_2.12-2.2.0
3. zookeeper-3.4.14
4. elasticsearch-6.3.1
5. kibana-6.3.1
6. filebeat-6.3.1
filebeat.prospectors:
type: log
enabled: true
paths:
/var/log/\*.log
#============================= Filebeat modules ===============================
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/\*.yml
# Set to true to enable config reloading
reload.enabled: false
# Period on which files under path should be checked for changes
#reload.period: 10s
#==================== Elasticsearch template setting ==========================
setup.template.settings:
index.number\_of\_shards: 3
#index.codec: best\_compression
#\_source.enabled: false
#-------------------------- Kafka Output ----------------------------------
output.kafka:
# initial brokers for reading cluster metadata
hosts: ["kafka ip:9092"]
# message topic selection + partitioning
topic: 'test'
partition.round\_robin:
reachable\_only: false
required\_acks: 1
compression: gzip
max\_message\_bytes: 1000000
We have check the logs in the kafka there is no log on it. Can you please help to fix this issue.
---
### Author:  [@Debashis](https://discuss.elastic.co/u/Debashis)
#### Post date: [April 29, 2019, 7:10am UTC](https://discuss.elastic.co/t/we-are-not-receiving-the-logs-from-filebeat-to-kafka/178756/2 "2019-04-29T07:10:30Z")
@saravananveera, Kindly use \ to provide config file.
I want to ask you some question before give a answer
1. What is the console debug log of filebeat?
2. Your mentioned host is reachable from filebeat node or not? Do you have mentioned your host
name in /etc/hosts?
3. What is the output of below command ?
`# bin/kafka-console-consumer.sh --zookeeper localhost:2181 --bootstrap-server :9092 --topic test --from-beginning`
---
### Author:  [@saravananveera](https://discuss.elastic.co/u/saravananveera)
#### Post date: [April 29, 2019, 8:11am UTC](https://discuss.elastic.co/t/we-are-not-receiving-the-logs-from-filebeat-to-kafka/178756/3 "2019-04-29T08:11:08Z")
Hi Debashis
Thanks for your response,
can you find the below information .
==================================================================
**1. Filebeat Logs for your reference:**
2019-04-29T07:56:11.063Z INFO kafka/log.go:36 producer/leader/test/0 abandoning broker 0
2019-04-29T07:56:11.063Z INFO kafka/log.go:36 producer/broker/0 shut down
2019-04-29T07:56:11.163Z INFO kafka/log.go:36 client/metadata fetching metadata for [test] from broker 10.11.12.159:9092
2019-04-29T07:56:11.165Z INFO kafka/log.go:36 producer/broker/0 starting up
2019-04-29T07:56:11.165Z INFO kafka/log.go:36 producer/broker/0 state change to [open] on test/0
2019-04-29T07:56:11.165Z INFO kafka/log.go:36 producer/leader/test/0 selected broker 0
2019-04-29T07:56:11.171Z INFO kafka/log.go:36 producer/broker/0 maximum request accumulated, waiting for space
2019-04-29T07:56:11.375Z INFO kafka/log.go:36 Failed to connect to broker kafka12.159:9092: dial tcp: lookup kafka12.159 on 8.8.8.8:53: no such host
2019-04-29T07:56:11.375Z INFO kafka/log.go:36 producer/broker/0 state change to [closing] because dial tcp: lookup kafka12.159 on 8.8.8.8:53: no such host
2019-04-29T07:56:11.377Z INFO kafka/log.go:36 producer/broker/0 state change to [closing] because dial tcp: lookup kafka12.159 on 8.8.8.8:53: no such host
===================================================================
2. we can able to reach kafka server from filebeat, Also, We didn't configured any host name in /etc/hosts file.
===================================================================
3. I hope this helps.
[root@kafka12 kafka]# bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
> welcome to ELK
[root@kafka12 kafka]# bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
welcome to ELK
===================================================================
**I went through an error message that the zookeeper is not recognized. So I have replaced the "--zookeeper" from your command**
[root@kafka12 kafka]# bin/kafka-console-consumer.sh --zookeeper localhost:2181 --bootstrap-server localhost:9092 --topic test --from-beginning
zookeeper is not a recognized option
[root@kafka12 kafka]# bin/kafka-console-consumer.sh localhost:2181 --bootstrap-server localhost:9092 --topic test --from-beginning
welcome to ELK
---
### Author:  [@Debashis](https://discuss.elastic.co/u/Debashis)
#### Post date: [April 29, 2019, 8:58am UTC](https://discuss.elastic.co/t/we-are-not-receiving-the-logs-from-filebeat-to-kafka/178756/4 "2019-04-29T08:58:19Z")
@saravananveera,
From your shared filebeat log it is confirmed that your filebeat is unable to connect with the kafka for this reason no logs are getting forwarded to the kafka.
`2019-04-29T07:56:11.375Z INFO kafka/log.go:36 Failed to connect to broker kafka12.159:9092: dial tcp: lookup kafka12.159 on 8.8.8.8:53: no such host`
---
### Author:  [@saravananveera](https://discuss.elastic.co/u/saravananveera)
#### Post date: [April 29, 2019, 10:57am UTC](https://discuss.elastic.co/t/we-are-not-receiving-the-logs-from-filebeat-to-kafka/178756/5 "2019-04-29T10:57:22Z")
Hi @Debashis
Thanks you for your support, After enter the localhost entry (/etc/hosts)in kafka ip from filebeat server.it's working fine.
---
### Author:  [@Debashis](https://discuss.elastic.co/u/Debashis)
#### Post date: [April 29, 2019, 11:13am UTC](https://discuss.elastic.co/t/we-are-not-receiving-the-logs-from-filebeat-to-kafka/178756/6 "2019-04-29T11:13:17Z")
It's Good 👍 @saravananveera .
Kindly mark this discussion as "Solved"
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [May 27, 2019, 11:13am UTC](https://discuss.elastic.co/t/we-are-not-receiving-the-logs-from-filebeat-to-kafka/178756/7 "2019-05-27T11:13:22Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.