Unstable indexing rate using S3 input plugin

I am using logstash to ingest json files from s3 to elasticsearch and our current indexing speed does not match my needs.
I noticed that the indexing rate dips regularly to 0. Sometimes for up to 5 minutes. And this pattern of dips mathes the network I/O metrics of the pod of logstash so my first guess would be that logstash does not download the s3 files while processing them?

But even so, the peak speed is not near my needs and the cpu/ram usage of logstash are far from 100% (more like around 40%).

What I have tried:

  • increasing the pipeline.workers
  • increasing the pipeline.batch.size
  • increasing the JVM heap size

I am pretty sure that the Elastic Search is not the bottle neck here.
If I forgot some informations I would be happy to provide them.

Thanks in advance for the help :slight_smile:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.