# Rate\_1min keeps dropping for my kafka to s3 batching on logstash

**URL:** <https://discuss.elastic.co/t/rate-1min-keeps-dropping-for-my-kafka-to-s3-batching-on-logstash/283508>\
**Category:** Logstash\
**Created:** [September 7, 2021, 8:45am UTC](https://discuss.elastic.co/t/rate-1min-keeps-dropping-for-my-kafka-to-s3-batching-on-logstash/283508 "2021-09-07T08:45:22Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Varsha1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/varsha1/32/78067_2.png) [@Varsha1](https://discuss.elastic.co/u/Varsha1)\
**Post date:** [September 7, 2021, 8:45am UTC](https://discuss.elastic.co/t/rate-1min-keeps-dropping-for-my-kafka-to-s3-batching-on-logstash/283508/1 "2021-09-07T08:45:22Z")

</div>

I'm trying to transfer data from kafka to s3, without any data transformation filters.

I added metrics filter plugin to calculate the rate of transfer, which keeps falling from ~16k to as low as ~3k. Is this expected and why is this happening.

Also, is there a documentation explaining how pipeline workers, pipeline batch size works and could be optimized?

I set mine to -w 250 -b 10000, and was able to transfer about ~2.2 million records in 15mins.  
I also used parameters on s3 output plugin -

```auto
upload_workers_count => 4
upload_queue_size => 10

```

But, I'm not sure how this works in conjunction with pipeline workers internally.

Please let me know if you'd like me to share more details.

```auto
rate: 16443.366747324828
rate: 16443.366747324828
rate: 15599.790007050075
rate: 14922.231184494047
rate: 14235.330257200087
rate: 13613.057503703576
rate: 13166.932721827068
rate: 12690.96250138113
rate: 12101.117155208596
rate: 11589.0722192815
rate: 11290.68819146028
rate: 10868.323755898386
rate: 10588.085525315939
.
.
.
rate: 9280.097099907236
rate: 9070.541738087844
rate: 8914.64899609739
rate: 8689.42618571362
rate: 8689.42618571362
rate: 8564.853290012046
rate: 8146.02568188926
rate: 7797.7573728437155
.
.
.
rate: 3638.8456399292595
rate: 2144.1724460065593
rate: 4356.41396880408
rate: 4108.3434015354505

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 5, 2021, 8:46am UTC](https://discuss.elastic.co/t/rate-1min-keeps-dropping-for-my-kafka-to-s3-batching-on-logstash/283508/2 "2021-10-05T08:46:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.