FileBeats with Redis

Elastic Stack Beats
1

Hi

My Current setup is

  • File Beat --> Log Stash ---> Elastic Search

The expected log volume is say 10 GB per hour to LS (as current version of FB doesn't have regexp feature and ships all logs).

Hence would like to know if FB and Log Stash would be able to handle this log without loss of any messages or network over head !!

What is the best practice !!
Should we implement FB-->Any Queue (Redis) --> Log Stash. If so does File beat support integrating with Fedis.

Please suggest.

2

We are planning to add support in the next release of Filebeat for regexp, that will reduce the number of logs. I'll come back to you when we finish with the implementation, so maybe you want to try one of our nightlies before the release date.

3

Hi Monica
Thanks. Would wait for the next release.

But my question is more on the architecture - whether my current setup could handle the load or need any intermediary Queuing system to ensure the load is balanced or throttled.

Not sure if this is the correct category for this query..

4

Redis output is deprecated and we encourage our Filebeat users to send data directly to Logstash. In 3.x release, we are planning to add persistent queuing in Logstash (https://github.com/elastic/logstash/issues/2605) and an additional queuing system will not be needed anymore.

2 Likes
5

Here is the link to the GitHub issue implementing regexp in Filebeat.

7

I'm getting this error message in Logstash 2.3:

"Beats input: the pipeline is blocked, temporary refusing new connection."

I'm assuming this is because I do not have a message queue between filebeat and logstash..

When you say "redis output is depricated, and we should send directly to logstash." What version of logstash does that pertain to?

8

totally unrelated to your problem. Please create a new topic. Message queue is optional + redis has been renewed for 5.x release (complete rewrite). Logstash problem due to pipeline in logstash being blocked by output or very slow filter.

9

Hey @monica: Is the persistent queuing in Logstash available now?

I am planning to use Redis in between Filebeat and Logstash.

10

It's a beta feature in 5.2. https://www.elastic.co/guide/en/logstash/5.2/persistent-queues.html