I have setup where logstash reads Kubernetes logs from 20 different buckets and send them to ELK. The logs seems to be coming 3-5 minutes late to ELK. The logstash running docker on VM with 31GB Xms/Xmx.
I am using one pipeline . Tried 6 pipelines with 2~3 each and got double/triple events from each pipeline.
How to speed up logstash ingestion from S3 buckets.
The buckets may had lots of files initially but now we have regular number of files. I have delete => true to delete file post processing. I have 10 cpus assigned to VM. I started with 16GB for jvm then kept increasing it.