# Logstash Architecture at Scale

**URL:** <https://discuss.elastic.co/t/logstash-architecture-at-scale/42495>\
**Category:** Logstash\
**Created:** [February 23, 2016, 1:42pm UTC](https://discuss.elastic.co/t/logstash-architecture-at-scale/42495 "2016-02-23T13:42:01Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![misterjulien](https://avatars.discourse-cdn.com/v4/letter/m/9de0a6/32.png) [@misterjulien](https://discuss.elastic.co/u/misterjulien)\
**Post date:** [February 23, 2016, 1:42pm UTC](https://discuss.elastic.co/t/logstash-architecture-at-scale/42495/1 "2016-02-23T13:42:01Z")

</div>

Regarding the doc "Deploying and Scaling Logstash", what would be an ideal way(s) to implement a mature Logstash cluster?

The picture at the bottom of the doc shows multiple instances of Shippers, a message queue, and one instance of an Indexer. Should each instance be in its own server? (individual shippers, queue, and indexer) Or should the message queue and indexer be on the same server? Can/Should you have multiple Shippers on the same server?

In my scenario I have a syslog-ng server and a load balancer that gets 50+ GB/day. I know that it will be more than one server can handle, so I'll be needing multiple servers to handle it.

Also, what's a good way to monitor the performance of the Logstash cluster? (events per second, dropped messages in the queue, memory use, etc)

Thanks in advance for your help.

-Eric

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [February 23, 2016, 6:49pm UTC](https://discuss.elastic.co/t/logstash-architecture-at-scale/42495/2 "2016-02-23T18:49:45Z")

</div>

> [@misterjulien](#):
>
> Should each instance be in its own server? (individual shippers, queue, and indexer) Or should the message queue and indexer be on the same server? Can/Should you have multiple Shippers on the same server?

Up to you and your resources really. If you aren't finding a single instance uses all the resources on the box then start another!

> [@misterjulien](#):
>
> Also, what's a good way to monitor the performance of the Logstash cluster? (events per second, dropped messages in the queue, memory use, etc)

You can use the metrics filter, but at the moment we don't have any monitoring visibility into LS. It's coming very soon though!

---

<div class="post-metadata">

**Author:** ![misterjulien](https://avatars.discourse-cdn.com/v4/letter/m/9de0a6/32.png) [@misterjulien](https://discuss.elastic.co/u/misterjulien)\
**Post date:** [February 24, 2016, 6:46pm UTC](https://discuss.elastic.co/t/logstash-architecture-at-scale/42495/3 "2016-02-24T18:46:36Z")

</div>

I get that it's up to me as to how I want to set it up, but I would imagine that there is a preferred/tested way that is better than others. If anyone has setup Logstash in the scenario where LS will be "drinking from a firehose", can you share what worked and what didn't work?

I'd rather not reinvent the wheel trying to optimize LS for a high volume feed.

Thanks.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:09am UTC](https://discuss.elastic.co/t/logstash-architecture-at-scale/42495/4 "2017-07-06T05:09:51Z")

</div>