# MAX TPS Achieved by file beat to Azure Event Hub : 600, anything more than that ends up in broken pipe **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [July 24, 2019, 6:58pm UTC](https://discuss.elastic.co/t/max-tps-achieved-by-file-beat-to-azure-event-hub-600-anything-more-than-that-ends-up-in-broken-pipe/192119 "2019-07-24T18:58:41Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Pragna\_Mohapatra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pragna_mohapatra/32/50850_2.png) [@Pragna\_Mohapatra](https://discuss.elastic.co/u/Pragna_Mohapatra)\ **Post date:** [July 24, 2019, 6:58pm UTC](https://discuss.elastic.co/t/max-tps-achieved-by-file-beat-to-azure-event-hub-600-anything-more-than-that-ends-up-in-broken-pipe/192119/1 "2019-07-24T18:58:41Z")
Hi, I am using kafka sink more to write the logs using file beat to as azure event hub. The maximum throughput I ever achieved is 700, anything beyond that just ends up broken pipes and my throughput drops drastically. I can't even work with more than 5 partitions. I get the following messages in a loop 2019-07-24T18:49:36.370Z INFO kafka/log.go:53 client/metadata fetching metadata for [trident-32] from broker \*.servicebus.windows.net:9093 2019-07-24T18:49:36.403Z INFO kafka/log.go:53 producer/leader/trident-32/22 selected broker 0 2019-07-24T18:49:36.403Z INFO kafka/log.go:53 producer/leader/trident-32/10 selected broker 0 2019-07-24T18:49:36.403Z INFO kafka/log.go:53 producer/broker/0 state change to [open] on trident-32/10 2019-07-24T18:49:36.403Z INFO kafka/log.go:53 producer/leader/trident-32/22 state change to [flushing-2] 2019-07-24T18:49:36.403Z INFO kafka/log.go:53 producer/leader/trident-32/10 state change to [flushing-2] 2019-07-24T18:49:36.403Z INFO kafka/log.go:53 producer/leader/trident-32/22 state change to [normal] 2019-07-24T18:49:36.503Z INFO kafka/log.go:53 producer/leader/trident-32/20 selected broker 0 2019-07-24T18:49:36.541Z INFO kafka/log.go:53 kafka message: Successful SASL handshake My filebeat configuration is: filebeat.inputs: - input\_type: log paths: - /mnt/trident/logs/test\*.log output.kafka: topic: trident-32 required\_acks: 1 client\_id: filebeat version: '1.0.0' hosts: - "_.servicebus.windows.net:9093" ssl.enabled: true username: "$ConnectionString" password: "Endpoint=sb://_.servicebus.windows.net/;SharedAccessKeyName=accessAll;SharedAccessKey=\*" bulk\_max\_size: 10000 worker: 33 compression: none partition.round\_robin: reachable\_only: true logging.level: INFO processors: - decode\_json\_fields: fields: ["message"] process\_array: false max\_depth: 1 target: "Trident" overwrite\_keys: false Is there anything I can change in my settings to keep the connection up and health without the broken pipe error. 2019-07-24T18:52:44.661Z INFO kafka/log.go:53 Connected to broker at \*.servicebus.windows.net:9093 (registered as #0) 2019-07-24T18:52:44.742Z INFO kafka/log.go:53 producer/broker/0 state change to [closing] because write tcp 172.16.6.6:42942-\>40.112.242.0:9093: write: connection reset by peer 2019-07-24T18:52:44.742Z INFO kafka/log.go:53 Error while closing connection to broker \*.servicebus.windows.net:9093: write tcp 172.16.6.6:42942-\>40.112.242.0:9093: write: broken pipe 2019-07-24T18:52:43.405Z INFO kafka/log.go:53 kafka message: Successful SASL handshake 2019-07-24T18:52:43.438Z INFO kafka/log.go:53 SASL authentication successful with broker \*.servicebus.windows.net:9093:4 - [0 0 0 0] 2019-07-24T18:52:43.438Z INFO kafka/log.go:53 Connected to broker at \*.servicebus.windows.net:9093 (registered as #0) 2019-07-24T18:52:43.546Z INFO kafka/log.go:53 producer/broker/0 state change to [closing] because write tcp 172.16.6.6:42922-\>40.112.242.0:9093: write: connection reset by peer 2019-07-24T18:52:43.546Z INFO kafka/log.go:53 Error while closing connection to broker \*.servicebus.windows.net:9093: write tcp 172.16.6.6:42922-\>40.112.242.0:9093: write: broken pipe ---
**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\ **Post date:** [July 25, 2019, 5:28pm UTC](https://discuss.elastic.co/t/max-tps-achieved-by-file-beat-to-azure-event-hub-600-anything-more-than-that-ends-up-in-broken-pipe/192119/2 "2019-07-25T17:28:32Z")
The reset by peer look like the remote host close the connection. How much TPS you get if you use the `bulk_max_size` and the `worker` because theses are pretty hight and would like to know the base performance. Is there any TPS limits on the Azure event hub? ---
**Author:** ![Pragna\_Mohapatra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pragna_mohapatra/32/50850_2.png) [@Pragna\_Mohapatra](https://discuss.elastic.co/u/Pragna_Mohapatra)\ **Post date:** [July 26, 2019, 3:33am UTC](https://discuss.elastic.co/t/max-tps-achieved-by-file-beat-to-azure-event-hub-600-anything-more-than-that-ends-up-in-broken-pipe/192119/3 "2019-07-26T03:33:40Z")
I could get a Max of 500 RPS. But but from last two days I am not able to get that either. We tried separately with 400 cores to load the event hub ... Throttling started around 33000 TPS ... ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 23, 2019, 3:33am UTC](https://discuss.elastic.co/t/max-tps-achieved-by-file-beat-to-azure-event-hub-600-anything-more-than-that-ends-up-in-broken-pipe/192119/4 "2019-08-23T03:33:41Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.