I'm trying to transfer data from kafka to s3, without any data transformation filters.
I added metrics filter plugin to calculate the rate of transfer, which keeps falling from ~16k to as low as ~3k. Is this expected and why is this happening.
Also, is there a documentation explaining how pipeline workers, pipeline batch size works and could be optimized?
I set mine to -w 250 -b 10000, and was able to transfer about ~2.2 million records in 15mins.
I also used parameters on s3 output plugin -
upload_workers_count => 4
upload_queue_size => 10
But, I'm not sure how this works in conjunction with pipeline workers internally.
Please let me know if you'd like me to share more details.
rate: 16443.366747324828
rate: 16443.366747324828
rate: 15599.790007050075
rate: 14922.231184494047
rate: 14235.330257200087
rate: 13613.057503703576
rate: 13166.932721827068
rate: 12690.96250138113
rate: 12101.117155208596
rate: 11589.0722192815
rate: 11290.68819146028
rate: 10868.323755898386
rate: 10588.085525315939
.
.
.
rate: 9280.097099907236
rate: 9070.541738087844
rate: 8914.64899609739
rate: 8689.42618571362
rate: 8689.42618571362
rate: 8564.853290012046
rate: 8146.02568188926
rate: 7797.7573728437155
.
.
.
rate: 3638.8456399292595
rate: 2144.1724460065593
rate: 4356.41396880408
rate: 4108.3434015354505