I have made a setup of ELK like below.
LS(reading from file) --> REDIS --> LS(indexer) --> ELS
I'm using a bench mark script from here to generate a 1M logs lines to measure performance.
- I'm monitoring redis list length with help of LLEN command
/usr/local/bin/redis-cli -h xxx.xxx.xxx.xxx LLEN "logstash-redis" - I'm monitoring the records in ELS with ruby script which uses stats page, which prints COUNT for every second.
What i have observed is that REDIS LIST is empty very quick almost in a ~65 seconds. I can only see only ~60% which ~600K records in ELS.
I'm trying to find the bottleneck here.
Does that mean 400K records are still in LS-indexer queue or in ELS ? How can i speed the things up ?
Is it possible its to do with performance of ES indexing ?