Put an intentional delay in event transmission to Logstash

Elastic Stack Beats
1

I have a use-case where I want to delay individual events being transmitted via Filebeat to Logstash. This is due to a race condition I'm encountering with another monitoring mechanism.

I've looked through the config options for Filebeat and haven't seen an option similar to this. Does anyone know if this functionality is possibly native to Filebeat or if it's on the development roadmap?

2

Hi,
Have you thought about using a message broker (Redis or alike) between Filebeat and Logstash in aid of some level of buffering?

3

I have considered using a message queue, however I'm trying to limit the number of services I have to rely on for this data pipeline.

Thanks for your suggestion. I suppose I'm pivoting towards creating a python script that will delay messages for 60s.

4

@seth.yes As well as scripting something timely, do consider the parameters that come with filebeat. close_inactive or scan_frequency. Backoff and harvester_buffer_size may be worth reading and testing with as well. All documented here: https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html

5

Thanks for the suggestion. I did review the capabilities of Filebeat -- I appreciate you bringing these options up but none of them really fit my use-case.

6

Can you share some more details on what you try to do? Do you try to "delay" a message, meaning if a line is read at time x, it should be sent 2s later. Or are you trying to throttle a limit the number of events? Or something else?

7

I'm simply trying to put a delay in a message: Once any one message is received, I'd like Logstash to sleep on pushing that one message through the filter for 60 seconds. Any one message would then be passed through my Logstash filter approximately 60 seconds after it was put into the file being tailed by Filebeat.

8

The closest thing I can think of is the sleep filter in logstash.

9

Sleep filter definitely does not do what I need. Sleep filter pauses the entire pipeline for the specified duration, effectively blocking any throughput.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.