# Filebeat to lagstash to kafka or Filebeat to kafka for \>= 10,000 servers

**URL:** https://discuss.elastic.co/t/filebeat-to-lagstash-to-kafka-or-filebeat-to-kafka-for-10-000-servers/88138
**Category:** Beats
**Tags:** filebeat
**Created:** [June 2, 2017, 10:36pm UTC](https://discuss.elastic.co/t/filebeat-to-lagstash-to-kafka-or-filebeat-to-kafka-for-10-000-servers/88138 "2017-06-02T22:36:16Z")
**Posts on this page:** 4
**Page:** 1

<div class="post-metadata">

### Author: ![amarc](https://avatars.discourse-cdn.com/v4/letter/a/ecd19e/32.png) [@amarc](https://discuss.elastic.co/u/amarc)
#### Post date: [June 2, 2017, 10:36pm UTC](https://discuss.elastic.co/t/filebeat-to-lagstash-to-kafka-or-filebeat-to-kafka-for-10-000-servers/88138/1 "2017-06-02T22:36:16Z")

</div>

Hi,  
We have a project to fingerprint logs from 10,000 servers and I am in process of designing a system for it.  
Looking at the design guide [https://www.elastic.co/blog/just-enough-kafka-for-the-elastic-stack-part1](https://www.elastic.co/blog/just-enough-kafka-for-the-elastic-stack-part1)  
Filebeat in version 5.x would support writing directly to Kafka. Do you think this is the better way than letting the filebeat clients write to logstash to kafka and then let logstash extract from kafka and inject into elasticsearch ?

---

<div class="post-metadata">

### Author: ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)
#### Post date: [June 6, 2017, 10:06am UTC](https://discuss.elastic.co/t/filebeat-to-lagstash-to-kafka-or-filebeat-to-kafka-for-10-000-servers/88138/2 "2017-06-06T10:06:42Z")

</div>

as you want to forward logs via kafka anyways, the only good reason for having beats-\>logstash-\>kafka is, if this logstash instance is going to modify the events to be written.

---

<div class="post-metadata">

### Author: ![amarc](https://avatars.discourse-cdn.com/v4/letter/a/ecd19e/32.png) [@amarc](https://discuss.elastic.co/u/amarc)
#### Post date: [June 9, 2017, 5:31pm UTC](https://discuss.elastic.co/t/filebeat-to-lagstash-to-kafka-or-filebeat-to-kafka-for-10-000-servers/88138/3 "2017-06-09T17:31:52Z")

</div>

@steffens  
Yup, logstash is going to modify the logs.  
I will be testing beats soon in our environment and if all goes well (resource utilization), I will be deploying it to few thousand servers. Could be an interesting use case for other folks here.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 7, 2017, 5:32pm UTC](https://discuss.elastic.co/t/filebeat-to-lagstash-to-kafka-or-filebeat-to-kafka-for-10-000-servers/88138/4 "2017-07-07T17:32:14Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
