Please consider the following input plugin configuration of Logstash:
host => "es1"
port => 6380
batch_count => 2000
data_type => "list"
key => "log_list"
codec => plain
I am trying to understand how does a redis input plugin of Logstash reads log files from Redis. I specifically want to know how does Logstash gets notified when a new log is available to be read or does it run a cron job to read log files from Redis server? Also, what does "list" data type and batch_count signifies as to how the data is read?
I have these confusions, and would be grateful if someone can provide some clarification please.
For a list, if batch_count is greater than one then the code sits in a loop making EVALSHA calls to a server side script. If the result set is empty then it does 'sleep 0.25'.
If batch_count is 1 then the code sits in a loop making BLPOP calls (which block until data is available or a timeout occurs).
If you use channel rather than list then code will subscribe to the channel. The subscribe call blocks and invokes the code block it is given each time a message arrives. The internals of how that works (polling, blocking etc. are a redis client question, not a logstash question).
@Badger Thank you so much for your reply. This is clear now. I have another query if you can answer please that would be great. We are saving log files on Redis server with the following name conventions:
Now, I wanted to know whether Redis stores the compressed files as it is (against a key, probably?)? And if so, then does Logstash reads each file at a time and pushes the logs to elasticsearch after parsing?
I have never actually used redis so I cannot answer that.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.