# Architecture suggestion

**URL:** <https://discuss.elastic.co/t/architecture-suggestion/126829>\
**Category:** Elasticsearch\
**Created:** [April 5, 2018, 2:02am UTC](https://discuss.elastic.co/t/architecture-suggestion/126829 "2018-04-05T02:02:13Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![las](https://avatars.discourse-cdn.com/v4/letter/l/5f9b8f/32.png) [@las](https://discuss.elastic.co/u/las)\
**Post date:** [April 5, 2018, 2:02am UTC](https://discuss.elastic.co/t/architecture-suggestion/126829/1 "2018-04-05T02:02:14Z")

</div>

I have a set of applications and have filebeat sending logs to logstash where they will be aggregated.

Is using logstash for aggregating logs of different format a good idea? As logs from multiple servers need to be aggregated, is it still possible to have more than one instance of logstash for horizontal scalability?

What are other options?

---

<div class="post-metadata">

**Author:** ![rcowart](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rcowart/32/88091_2.png) [@rcowart](https://discuss.elastic.co/u/rcowart)\
**Post date:** [April 5, 2018, 5:44am UTC](https://discuss.elastic.co/t/architecture-suggestion/126829/2 "2018-04-05T05:44:14Z")

</div>

For horizontal scalability I would have filebeat send the data to Kafka (more complex but more powerful) or redis (super simple, but only a simple queue). All beats have these outputs. You can then have multiple instances of Logstash which pickup the data from the queue.

---

<div class="post-metadata">

**Author:** ![las](https://avatars.discourse-cdn.com/v4/letter/l/5f9b8f/32.png) [@las](https://discuss.elastic.co/u/las)\
**Post date:** [April 5, 2018, 3:38pm UTC](https://discuss.elastic.co/t/architecture-suggestion/126829/3 "2018-04-05T15:38:09Z")

</div>

If I use Kafka for aggregating events from multiple filebeat, do I need to write custom consumers to parse the events or is there configuration driven plugins like grok etc for logstash?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 3, 2018, 3:38pm UTC](https://discuss.elastic.co/t/architecture-suggestion/126829/4 "2018-05-03T15:38:14Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.