# Logstash S3 input slow ingestion **URL:** \ **Category:** Logstash\ **Tags:** docker\ **Created:** [March 14, 2023, 12:15pm UTC](https://discuss.elastic.co/t/logstash-s3-input-slow-ingestion/327653 "2023-03-14T12:15:58Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![true64gurus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/true64gurus/32/107237_2.png) [@true64gurus](https://discuss.elastic.co/u/true64gurus)\ **Post date:** [March 14, 2023, 12:15pm UTC](https://discuss.elastic.co/t/logstash-s3-input-slow-ingestion/327653/1 "2023-03-14T12:15:58Z")
I have setup where logstash reads Kubernetes logs from 20 different buckets and send them to ELK. The logs seems to be coming 3-5 minutes late to ELK. The logstash running docker on VM with 31GB Xms/Xmx. I am using one pipeline . Tried 6 pipelines with 2~3 each and got double/triple events from each pipeline. How to speed up logstash ingestion from S3 buckets. Thanks ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [March 14, 2023, 12:53pm UTC](https://discuss.elastic.co/t/logstash-s3-input-slow-ingestion/327653/2 "2023-03-14T12:53:05Z")
> [@true64gurus](#): > > I have setup where logstash reads Kubernetes logs from 20 different buckets and send them to ELK Do these buckets have a lot of files in them? > [@true64gurus](#): > > The logstash running docker on VM with 31GB Xms/Xmx. This seems way too much, Logstash is more CPU bound than memory bound, what is the CPU count for your Logstash container? ---
**Author:** ![true64gurus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/true64gurus/32/107237_2.png) [@true64gurus](https://discuss.elastic.co/u/true64gurus)\ **Post date:** [March 14, 2023, 1:33pm UTC](https://discuss.elastic.co/t/logstash-s3-input-slow-ingestion/327653/3 "2023-03-14T13:33:46Z")
The buckets may had lots of files initially but now we have regular number of files. I have delete =\> true to delete file post processing. I have 10 cpus assigned to VM. I started with 16GB for jvm then kept increasing it. Steady state shows ~ 80-90 MB per bucket. ---
**Author:** ![true64gurus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/true64gurus/32/107237_2.png) [@true64gurus](https://discuss.elastic.co/u/true64gurus)\ **Post date:** [March 14, 2023, 2:41pm UTC](https://discuss.elastic.co/t/logstash-s3-input-slow-ingestion/327653/4 "2023-03-14T14:41:19Z")
@leandrojmp do I need to change the way kubernetes clusters saves logs to buckets to lower number of files ? Right now files get saved on each bucket as "cluster\_name/yyyy/mm/dd/yyyymmddxxxxxx\_\_yy.gz" ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [March 14, 2023, 3:38pm UTC](https://discuss.elastic.co/t/logstash-s3-input-slow-ingestion/327653/5 "2023-03-14T15:38:35Z")
If you can reduce the number of files I think that you should try to do it. Logstash s3 input has a couple of issues when working with buckets with a lot of files. Personally I do not use this input because the performance is pretty bad in my use case (logs from AWS services) and I was not able to fix or improve it, so a custom collector was needed. ---
**Author:** ![true64gurus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/true64gurus/32/107237_2.png) [@true64gurus](https://discuss.elastic.co/u/true64gurus)\ **Post date:** [March 14, 2023, 3:55pm UTC](https://discuss.elastic.co/t/logstash-s3-input-slow-ingestion/327653/6 "2023-03-14T15:55:12Z")
@leandrojmp what do you recommend for temporary storage for Kubernetes logs until logstash pulls them. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 11, 2023, 3:55pm UTC](https://discuss.elastic.co/t/logstash-s3-input-slow-ingestion/327653/7 "2023-04-11T15:55:50Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.