I have a logstash agent that extracts data from redis, preprocesses it and reengines it in redis again.
I have been able to incorporate in this agent the metric filter to give me an idea of how fast the process is.
Approximately, I have an average yield of 4000 processed trails per second.
The logstash version I work with in 5.1.1 and the redis version 3.2.5.
The installation I made of logstash is intended to be deployed several agents of the same type within an agent directory, so that the same instance of logstash raise N agents, as many as I declare within the agents directory.
The problem I have is that I do not see that it is increasing at all the speed of process of logstash increasing the number of agents of logstash in the instance that I raise.
The same is an error in my perception of how the performance increase works.
As part of improving the logstash agent itself to make it faster, if I raise more "equal" agents in the same instance, do I expect a global increase in processing?
Redis is single threaded. Each client request to Redis will be handled serially.
The logstash redis input can read in larger batches. It uses a LUA script to read a batch. In Redis only one LUA script can run at a time.
The redis output writes in batches too.
You need to understand the how much time it takes for the Logstash(es) feeding data to Redis take to build up their batch and how long it takes for the redis output to write and the same for the Logstash(es) reading data from Redis.
For best performance a good interleave of writes and reads must be seen.
______________~~~~~~~~~~~~~~~~~~~~~~~____~~~~~~~~~~~~~______~~~~~~~~~~~~~~~~~~~~~~~____________
Logstash writer 1 | build batch |write| | build batch |write| | build batch |write|
Logstash writer 2 | build batch |write| | build batch |write| | build batch |write|
Logstash reader 1 |read| process batch + ES out | |read| process batch + ES out|
Logstash reader 2 |read| process batch + ES out| |read| process batch + ES out|
In the above ascii diagram, the read and write durations are small - if build batch or process batch + ES out durations are much longer than read or write then you should get few waits in Redis.
Obviously the more LS instances you have (redis clients really) with simple filter and output sections the more waits at Redis you will have.
I think if you use Kafka then you have to worry less about parallel reads and writes.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
