Trying to understand how Logstash Redis input plugin works

[ImranArif]

Hi,
Please consider the following input plugin configuration of Logstash:

input {
  redis {
    host => "es1"
    port => 6380
    batch_count => 2000
    data_type => "list"
    key => "log_list"
    codec => plain
  }
}

I am trying to understand how does a redis input plugin of Logstash reads log files from Redis. I specifically want to know how does Logstash gets notified when a new log is available to be read or does it run a cron job to read log files from Redis server? Also, what does "list" data type and batch_count signifies as to how the data is read?

I have these confusions, and would be grateful if someone can provide some clarification please.

[Badger]

For a list, if batch_count is greater than one then the code sits in a loop making EVALSHA calls to a server side script. If the result set is empty then it does 'sleep 0.25'.

If batch_count is 1 then the code sits in a loop making BLPOP calls (which block until data is available or a timeout occurs).

If you use channel rather than list then code will subscribe to the channel. The subscribe call blocks and invokes the code block it is given each time a message arrives. The internals of how that works (polling, blocking etc. are a redis client question, not a logstash question).

[ImranArif]

@Badger Thank you so much for your reply. This is clear now. I have another query if you can answer please that would be great. We are saving log files on Redis server with the following name conventions:

911810C4B60D28180721B9C20CD0108D_20220825_06_caa_001.log.gz
911810C4B60D28180721B9C20CD0108D_20220825_06_caa_002.log.gz

Now, I wanted to know whether Redis stores the compressed files as it is (against a key, probably?)? And if so, then does Logstash reads each file at a time and pushes the logs to elasticsearch after parsing?

[Badger]

I have never actually used redis so I cannot answer that.