At what point should I use a queue

Elastic Stack Logstash
1

I have been working on just an ELK stack with LSF. But I have come to the cross roads of do I need to put a queue in my system? I have only been on a test environment and do not even know what type of log traffic to expect when in production, its up to the client. At what traffic level should I start to put in a queue? 100/sec, 1000/sec, 10000/sec? And is there a way to test if logs are actually dropping?

Thanks.
M

2

How many Logstash machines do you have? One point of using a broker is that all Logstash instances can pull messages from the broker. If the log clients are pushing messages the distribution of new events between instances will be worse.

LSF uses TCP and will back off if Logstash can't accept the messages. Therefore you should not lose any messages unless log files are rotated out of existence while LSF is backed up. Of course, if Logstash is clogged for an extended period you might run into problems with a broker too.

3

I am looking at initial start of 2 logstash machines. I find this a problem but its what I was given: 1 blade with 2 vms on it each which will have its own LS. But I will try to find some more details on what hardware I have to use.

I think I am understanding this more! So broker is just trying to distribute as evenly as possible to many LS instances. But does the throughput or messages/sec determine the requirement of a broker? For some reason I am finding this misleading in so many chats and blogs that brokers are needed for a single reason, high volume.

I might be wrong but I found this on git that LSF or I am using log courier that downtime is not a problem with rotation. Meaning if LS is bottled up or shutdown LSF will keep the file open. Any new rotations will be read and kept open. Once LS frees up or comes online it will resume. I should probably read the source code or trace the code to see how this really works! Very interesting stuff.

Thanks,
M

4

So broker is just trying to distribute as evenly as possible to many LS instances. But does the throughput or messages/sec determine the requirement of a broker? For some reason I am finding this misleading in so many chats and blogs that brokers are needed for a single reason, high volume.

If you have high volume you're probably interested in distributing the load, so one doesn't preclude the other.