New to ELK, do we need a queue for ELK stack

Elastic Stack Logstash
1

Thanks

2

No, a queue isn't required.

3

How to decide if i need a queue or not?

Can anyone share an example ?

4

One example of when you might benefit from introducing a queue of some kind is then you have inputs that do not handle back-pressure well and you will need to be able to buffer events within the pipeline in order to not lose data if a part of the pipeline slows down or stops. This can be due to unreliable connections between data centres or Elasticsearch having issues and not taking reads.

Queues can also help decouple collection from indexing and allow you to scale these layers independently.

1 Like
5

I am using filebeat as input and input_type is log. how many logstash and elasticsearch do I need to make sure elk stack can perform well. and also how to decide if queue is necessary for the stack?

6

I am using filebeat as input and input_type is log.

Filebeat handles backpressure so there's no urgent need for a queue.

how many logstash and elasticsearch do I need to make sure elk stack can perform well.

That obviously depends on the load you're putting on the system. Start small and increase traffic slowly. Employ monitoring to see how well it performs.

and also how to decide if queue is necessary for the stack?

I think @Christian_Dahlqvist has answered this.

7

For monitoring ELK, any suggestion for implementing it.

For elasticsearch, there are plugins to do it like marvel.