Hello!
We faced some issues with our Redis & Logstash setup.
syslog_data --> |->rsyslog->redis->logstash->| --> elastic'
RSyslog & Redis & Logstash -- are installed on one physical server (8x cores,). I try to improve and analyze performance of the setup.
In total, we receive ~22K EPS from our firewalls. I disabled all the filters on logstash. So, in the clear environment we have the 22K EPS.
But in this case I saw the CPU usage by redis input thread:
23251 logstash 39 19 23.116g 782196 11232 S 70.0 2.4 5:03.23 [main]<redis
23250 logstash 39 19 23.116g 782196 11232 S 68.3 2.4 5:03.43 [main]<redis
23255 logstash 39 19 23.116g 782196 11232 R 66.3 2.4 5:07.66 [main]<redis
23252 logstash 39 19 23.116g 782196 11232 S 65.3 2.4 5:02.04 [main]<redis
23249 logstash 39 19 23.116g 782196 11232 R 63.0 2.4 5:10.02 [main]<redis
23248 logstash 39 19 23.116g 782196 11232 S 62.0 2.4 5:03.48 [main]<redis
23253 logstash 39 19 23.116g 782196 11232 R 55.3 2.4 5:02.23 [main]<redis
23254 logstash 39 19 23.116g 782196 11232 S 48.3 2.4 4:59.23 [main]<redis
It looks pretty high to me. Our current input config:
input {
redis {
data_type => "list"
key => "rsyslog_logstash"
batch_count => 125
type => syslog
codec => "json"
threads => 8
}
}
Is it normal situation? I'm afraid that cpu usage may impact the performance with enabled filters.
Thanks!