# Filebeat output to Kafka, how do I do stack monitoring

**URL:** <https://discuss.elastic.co/t/filebeat-output-to-kafka-how-do-i-do-stack-monitoring/271209>\
**Category:** Beats\
**Tags:** elastic-stack-monitoring, filebeat\
**Created:** [April 26, 2021, 2:26am UTC](https://discuss.elastic.co/t/filebeat-output-to-kafka-how-do-i-do-stack-monitoring/271209 "2021-04-26T02:26:10Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![ktpktr0](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ktpktr0/32/87157_2.png) [@ktpktr0](https://discuss.elastic.co/u/ktpktr0)\
**Post date:** [April 26, 2021, 2:26am UTC](https://discuss.elastic.co/t/filebeat-output-to-kafka-how-do-i-do-stack-monitoring/271209/1 "2021-04-26T02:26:10Z")

</div>

I output the log to Kafka and how I add filebeat to the stack monitor. I try the following, and it works well, but on kibana, I still don't see the state of filebeat.

```auto
output.kafka:
  hosts: ["192.168.1.190:9092"]
  topic: "%{[kafka_topic]}"
  required_acks: 1
  username: "producer"
  password: "xxx"

logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat.log
  keepfiles: 3
  permissions: 0644

# ES监控，可选功能
monitoring:
  enabled: true
  elasticsearch:
    hosts: ["https://192.168.1.191:9200"]
    metrics.period: 120
    backoff.max: 180
    state.period: 600
    backoff.init: 10
    username: "xxx"
    password: "xxxx"
    ssl.certificate_authorities: /etc/filebeat/ssl/ca.crt

```

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/9/89476dc21b9d374cf9013a2678917a7aaccf08ee.png)

---

<div class="post-metadata">

**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\
**Post date:** [April 27, 2021, 1:16am UTC](https://discuss.elastic.co/t/filebeat-output-to-kafka-how-do-i-do-stack-monitoring/271209/2 "2021-04-27T01:16:41Z")

</div>

I believe that is for elasticsearch server logs, as in a filebeat instance parsing filebeat logs. There should be a separate section in the stack monitoring to see the beats that are pushing to elasticsearch.

---

<div class="post-metadata">

**Author:** ![ktpktr0](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ktpktr0/32/87157_2.png) [@ktpktr0](https://discuss.elastic.co/u/ktpktr0)\
**Post date:** [April 27, 2021, 1:32am UTC](https://discuss.elastic.co/t/filebeat-output-to-kafka-how-do-i-do-stack-monitoring/271209/3 "2021-04-27T01:32:48Z")

</div>

You mean log collection and stack monitoring are two separate parts? Is my configuration correct

---

<div class="post-metadata">

**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\
**Post date:** [April 27, 2021, 1:59am UTC](https://discuss.elastic.co/t/filebeat-output-to-kafka-how-do-i-do-stack-monitoring/271209/4 "2021-04-27T01:59:30Z")

</div>

So what you have configured will push filebeat metrics to to this ES server `https://192.168.1.191:9200`. That doesn't mean that the actual parsed logs will be sent to the same ES server. You have it being sent to Kafka, what are you doing with the log data after Kafka?

---

<div class="post-metadata">

**Author:** ![ktpktr0](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ktpktr0/32/87157_2.png) [@ktpktr0](https://discuss.elastic.co/u/ktpktr0)\
**Post date:** [April 27, 2021, 3:17am UTC](https://discuss.elastic.co/t/filebeat-output-to-kafka-how-do-i-do-stack-monitoring/271209/5 "2021-04-27T03:17:29Z")

</div>

I can see the monitoring of filebeat from this entrance, but I don't know why it's not in the cluster below. I will send the log to Kafka, how to configure the monitoring of filebeat.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/c/1c189ae2b9986676dd7c83949bdcdd09e60a1dc5.png)

---

<div class="post-metadata">

**Author:** ![ktpktr0](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ktpktr0/32/87157_2.png) [@ktpktr0](https://discuss.elastic.co/u/ktpktr0)\
**Post date:** [April 27, 2021, 7:39am UTC](https://discuss.elastic.co/t/filebeat-output-to-kafka-how-do-i-do-stack-monitoring/271209/6 "2021-04-27T07:39:55Z")

</div>

Thanks for your help, I added cluster in filebeat configuration\_ After UUID, filebeat can be monitored normally

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 25, 2021, 9:40am UTC](https://discuss.elastic.co/t/filebeat-output-to-kafka-how-do-i-do-stack-monitoring/271209/7 "2021-05-25T09:40:34Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.