# New to ELK, do we need a queue for ELK stack

**URL:** <https://discuss.elastic.co/t/new-to-elk-do-we-need-a-queue-for-elk-stack/48829>\
**Category:** Logstash\
**Created:** [April 29, 2016, 5:10pm UTC](https://discuss.elastic.co/t/new-to-elk-do-we-need-a-queue-for-elk-stack/48829 "2016-04-29T17:10:48Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![nanshan](https://avatars.discourse-cdn.com/v4/letter/n/858c86/32.png) [@nanshan](https://discuss.elastic.co/u/nanshan)\
**Post date:** [April 29, 2016, 5:10pm UTC](https://discuss.elastic.co/t/new-to-elk-do-we-need-a-queue-for-elk-stack/48829/1 "2016-04-29T17:10:48Z")

</div>

Thanks

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 30, 2016, 2:55pm UTC](https://discuss.elastic.co/t/new-to-elk-do-we-need-a-queue-for-elk-stack/48829/2 "2016-04-30T14:55:36Z")

</div>

No, a queue isn't required.

---

<div class="post-metadata">

**Author:** ![nanshan](https://avatars.discourse-cdn.com/v4/letter/n/858c86/32.png) [@nanshan](https://discuss.elastic.co/u/nanshan)\
**Post date:** [June 16, 2016, 8:19pm UTC](https://discuss.elastic.co/t/new-to-elk-do-we-need-a-queue-for-elk-stack/48829/3 "2016-06-16T20:19:39Z")

</div>

How to decide if i need a queue or not?

Can anyone share an example ?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [June 16, 2016, 8:24pm UTC](https://discuss.elastic.co/t/new-to-elk-do-we-need-a-queue-for-elk-stack/48829/4 "2016-06-16T20:24:31Z")

</div>

One example of when you might benefit from introducing a queue of some kind is then you have inputs that do not handle back-pressure well and you will need to be able to buffer events within the pipeline in order to not lose data if a part of the pipeline slows down or stops. This can be due to unreliable connections between data centres or Elasticsearch having issues and not taking reads.

Queues can also help decouple collection from indexing and allow you to scale these layers independently.

---

<div class="post-metadata">

**Author:** ![nanshan](https://avatars.discourse-cdn.com/v4/letter/n/858c86/32.png) [@nanshan](https://discuss.elastic.co/u/nanshan)\
**Post date:** [June 20, 2016, 5:43pm UTC](https://discuss.elastic.co/t/new-to-elk-do-we-need-a-queue-for-elk-stack/48829/5 "2016-06-20T17:43:55Z")

</div>

I am using filebeat as input and input\_type is log. how many logstash and elasticsearch do I need to make sure elk stack can perform well. and also how to decide if queue is necessary for the stack?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 20, 2016, 8:06pm UTC](https://discuss.elastic.co/t/new-to-elk-do-we-need-a-queue-for-elk-stack/48829/6 "2016-06-20T20:06:15Z")

</div>

> I am using filebeat as input and input\_type is log.

Filebeat handles backpressure so there's no urgent need for a queue.

> how many logstash and elasticsearch do I need to make sure elk stack can perform well.

That obviously depends on the load you're putting on the system. Start small and increase traffic slowly. Employ monitoring to see how well it performs.

> and also how to decide if queue is necessary for the stack?

I think @Christian_Dahlqvist has answered this.

---

<div class="post-metadata">

**Author:** ![nanshan](https://avatars.discourse-cdn.com/v4/letter/n/858c86/32.png) [@nanshan](https://discuss.elastic.co/u/nanshan)\
**Post date:** [June 20, 2016, 8:16pm UTC](https://discuss.elastic.co/t/new-to-elk-do-we-need-a-queue-for-elk-stack/48829/7 "2016-06-20T20:16:53Z")

</div>

For monitoring ELK, any suggestion for implementing it.

For elasticsearch, there are plugins to do it like marvel.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:51am UTC](https://discuss.elastic.co/t/new-to-elk-do-we-need-a-queue-for-elk-stack/48829/8 "2017-07-06T04:51:45Z")

</div>


