I'm trying to transfer data from kafka to s3, without any data transformation filters.
I added metrics filter plugin to calculate the rate of transfer, which keeps falling from ~16k to as low as ~3k. Is this expected and why is this happening.
Also, is there a documentation explaining how pipeline workers, pipeline batch size works and could be optimized?
I set mine to -w 250 -b 10000, and was able to transfer about ~2.2 million records in 15mins.
I also used parameters on s3 output plugin -
upload_workers_count => 4 upload_queue_size => 10
But, I'm not sure how this works in conjunction with pipeline workers internally.
Please let me know if you'd like me to share more details.
rate: 16443.366747324828 rate: 16443.366747324828 rate: 15599.790007050075 rate: 14922.231184494047 rate: 14235.330257200087 rate: 13613.057503703576 rate: 13166.932721827068 rate: 12690.96250138113 rate: 12101.117155208596 rate: 11589.0722192815 rate: 11290.68819146028 rate: 10868.323755898386 rate: 10588.085525315939 . . . rate: 9280.097099907236 rate: 9070.541738087844 rate: 8914.64899609739 rate: 8689.42618571362 rate: 8689.42618571362 rate: 8564.853290012046 rate: 8146.02568188926 rate: 7797.7573728437155 . . . rate: 3638.8456399292595 rate: 2144.1724460065593 rate: 4356.41396880408 rate: 4108.3434015354505