Trying to understand how Logstash Redis input plugin works

Please consider the following input plugin configuration of Logstash:

input {
  redis {
    host => "es1"
    port => 6380
    batch_count => 2000
    data_type => "list"
    key => "log_list"
    codec => plain

I am trying to understand how does a redis input plugin of Logstash reads log files from Redis. I specifically want to know how does Logstash gets notified when a new log is available to be read or does it run a cron job to read log files from Redis server? Also, what does "list" data type and batch_count signifies as to how the data is read?

I have these confusions, and would be grateful if someone can provide some clarification please.

For a list, if batch_count is greater than one then the code sits in a loop making EVALSHA calls to a server side script. If the result set is empty then it does 'sleep 0.25'.

If batch_count is 1 then the code sits in a loop making BLPOP calls (which block until data is available or a timeout occurs).

If you use channel rather than list then code will subscribe to the channel. The subscribe call blocks and invokes the code block it is given each time a message arrives. The internals of how that works (polling, blocking etc. are a redis client question, not a logstash question).

1 Like

@Badger Thank you so much for your reply. This is clear now. I have another query if you can answer please that would be great. We are saving log files on Redis server with the following name conventions:


Now, I wanted to know whether Redis stores the compressed files as it is (against a key, probably?)? And if so, then does Logstash reads each file at a time and pushes the logs to elasticsearch after parsing?

I have never actually used redis so I cannot answer that.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.