i read the reference https://www.elastic.co/guide/en/logstash/current/deploying-and-scaling.html,
in the article,shipping logstash collect logs first then logs pass messge queue finally indexing logstash filter logs.
in our situation,we have many apps on different machines,we need to collect logs apps produced, our solution is that first we use logstash to filter logs then pass message queue finally use logstash ship logs to es. i think this solution can balance logstash filter pressure on machines and better than first ship then filter. is that right?thanks in advance!
Please format your post, it is impossible to read.
Why would it be easier to load balance before the message broker? Half the point of using a message broker is the ease with which one can balance the load.
The other main point is that you want to get your logs away from your critical path as soon as possible, and using the message broker as a buffer before the filters makes good sense since the first Logstash instance can accept messages at a high rate.
thanks you answer!
because indexing logstash could consume more resources than shipping logstash,our applications deploy on different server,if i first index then ship,the indexing pressure can balance on different server,on the contrast,there would be more pressure on indexing logstash ?
Okay, I see what you mean. It makes sense if you don't mind running a potentially expensive Logstash process on each server and if you're not interested in central filtering. I prefer lightweight shippers and centralized filtering.