# Using ELK to create a unified logging system with large log input

**URL:** <https://discuss.elastic.co/t/using-elk-to-create-a-unified-logging-system-with-large-log-input/53261>\
**Category:** Logstash\
**Created:** [June 20, 2016, 4:29am UTC](https://discuss.elastic.co/t/using-elk-to-create-a-unified-logging-system-with-large-log-input/53261 "2016-06-20T04:29:44Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![rvu95](https://avatars.discourse-cdn.com/v4/letter/r/f9ae1b/32.png) [@rvu95](https://discuss.elastic.co/u/rvu95)\
**Post date:** [June 20, 2016, 4:29am UTC](https://discuss.elastic.co/t/using-elk-to-create-a-unified-logging-system-with-large-log-input/53261/1 "2016-06-20T04:29:44Z")

</div>

Hello, I'm still a newbie, I want to ask about the tradeoff of using ELK for unified logging syste, with large log input. I'm planning on using HAProxy, redis, and ELK. What wanna I ask is how's the performance of this kind of system while the log rate of the system is large? Should I use more of redis cluster as a buffer to prevent the missing of the package?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [June 20, 2016, 4:44am UTC](https://discuss.elastic.co/t/using-elk-to-create-a-unified-logging-system-with-large-log-input/53261/2 "2016-06-20T04:44:13Z")

</div>

That depends on a lot of things.  
Ultimately need to scale to handle both high indexing and searching, and you should test based on your use case.

---

<div class="post-metadata">

**Author:** ![rvu95](https://avatars.discourse-cdn.com/v4/letter/r/f9ae1b/32.png) [@rvu95](https://discuss.elastic.co/u/rvu95)\
**Post date:** [June 20, 2016, 4:46am UTC](https://discuss.elastic.co/t/using-elk-to-create-a-unified-logging-system-with-large-log-input/53261/3 "2016-06-20T04:46:58Z")

</div>

how about the queueing mechanism that will be needed to handle the input that come when the ES performance is slowing down because of the large data that being inserted?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [June 20, 2016, 4:48am UTC](https://discuss.elastic.co/t/using-elk-to-create-a-unified-logging-system-with-large-log-input/53261/4 "2016-06-20T04:48:21Z")

</div>

If you are looking at very large scale then I'd suggest you consider kafka instead of redis.  
But, a broker is a solid option.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:52am UTC](https://discuss.elastic.co/t/using-elk-to-create-a-unified-logging-system-with-large-log-input/53261/5 "2017-07-06T04:52:00Z")

</div>