# Logstash: replacement of the Coralogix/Ruby output plugin with http output plugin requires x2 resources **URL:** \ **Category:** Logstash\ **Created:** [March 13, 2023, 1:46pm UTC](https://discuss.elastic.co/t/logstash-replacement-of-the-coralogix-ruby-output-plugin-with-http-output-plugin-requires-x2-resources/327558 "2023-03-13T13:46:27Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![AlexKonkin](https://avatars.discourse-cdn.com/v4/letter/a/eb9ed0/32.png) [@AlexKonkin](https://discuss.elastic.co/u/AlexKonkin)\ **Post date:** [March 13, 2023, 1:46pm UTC](https://discuss.elastic.co/t/logstash-replacement-of-the-coralogix-ruby-output-plugin-with-http-output-plugin-requires-x2-resources/327558/1 "2023-03-13T13:46:27Z")
According to the Coralogix they are going to drop support of their Coralogix/Ruby based plugin. As the replacement it is proposed to use http output plugin. You can find the relevant details by navigating the URL below: [Logstash - Coralogix](https://coralogix.com/docs/coralogix-logstash-integration/) The behaviour of the system before integration: The messages are supplied to the Kafka topic with 50 partitions (input). As the output was used the Coralogix/Ruby plugin and in was able to handle the workload with the 10 instances of the same pipeline which used the Coralogix/Ruby based plugin. The behavior of the system after the integration: 1.The successful case: Logstash pipeline can handle the workload of the production from the Kafka topic with 50 partitions with 20 pipelines (instead of 10 with Coralogix/Ruby) which in turn significantly increases the expences for this part of the infrastructure (ec2 c2.8xlarge - 992$ per month). The solutuin works but its price is 2 times higher than the original one (Coralogix/Ruby) You can find the sample of the output configuration below. I would like also to mention that for this setup the value of the pool\_max = 50 by default ```auto output { http { url => "${CORALOGIX_MAIN_API_URL}" http_method => "post" headers => ["private_key", "${CORALOGIX_MAIN_PRIVKEY}"] format => "json_batch" codec => "json" mapping => { "applicationName" => "${ENVIRONMENT}" "subsystemName" => "${SUBSYSTEM}" "text" => "%%{[@metadata][event]}" } http_compression => true automatic_retries => 5 retry_non_idempotent => false connect_timeout => 60 keepalive => true pool_max => 50 } } ``` Attempt to optimise the expences In attempt to optimise the expences (decrease a number of pipelines from 20) I tried to increase the pool\_max value to 300 during a set of probes and managed to handle the workload with a noticable lags for 4 or 5 hours, after this period I got a high Lag of non consumed messages and had to stop the test (13 pipelines were not able to handle the workload) ```auto output { http { url => "${CORALOGIX_MAIN_API_URL}" http_method => "post" headers => ["private_key", "${CORALOGIX_MAIN_PRIVKEY}"] format => "json_batch" codec => "json" mapping => { "applicationName" => "${ENVIRONMENT}" "subsystemName" => "${SUBSYSTEM}" "text" => "%%{[@metadata][event]}" } http_compression => true automatic_retries => 5 retry_non_idempotent => false connect_timeout => 60 keepalive => true pool_max => 300 } } ``` Question: Is this expected behavior for the http plugin? I mean the high number of logstash pipeline instances, supposedly equal or less equal to the number of the Kafka topic partitions? In my case Kafka topic partitions = 50 and 40 pipelines were able to cover the load instead of the 20 with Coralogix/Ruby. Is there any other/additional approach that could decrease a cost of this solution and a number of the logstash pipelines that handle the worload? Thank you. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [March 13, 2023, 2:38pm UTC](https://discuss.elastic.co/t/logstash-replacement-of-the-coralogix-ruby-output-plugin-with-http-output-plugin-requires-x2-resources/327558/2 "2023-03-13T14:38:52Z")
What is the pipeline configuration? Have you tried to increase the `pipeline.batch.size`? Also, what is the average event rate and event size? Per default the `pipeline.batch.size` is `125`, try to increase this size to see if it increases the throughput. ---
**Author:** ![AlexKonkin](https://avatars.discourse-cdn.com/v4/letter/a/eb9ed0/32.png) [@AlexKonkin](https://discuss.elastic.co/u/AlexKonkin)\ **Post date:** [March 15, 2023, 8:09am UTC](https://discuss.elastic.co/t/logstash-replacement-of-the-coralogix-ruby-output-plugin-with-http-output-plugin-requires-x2-resources/327558/3 "2023-03-15T08:09:47Z")
Hello, Thank you so much for your reply. I did not know about these settings and tried to apply adjustments only on a plugin level. I see that the retention period is 2 days, so I get the number of all messages and divided it by 2. According to the calculation above the rate of the messages per day is 4167026334 The rate of the messages per hour is 50.000 I used the default value for the `pipeline.batch.size` I will try to increase it and in case if I succeed with this setup I will update this thread. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [March 15, 2023, 3:27pm UTC](https://discuss.elastic.co/t/logstash-replacement-of-the-coralogix-ruby-output-plugin-with-http-output-plugin-requires-x2-resources/327558/4 "2023-03-15T15:27:19Z")
> [@AlexKonkin](#): > > According to the calculation above the rate of the messages per day is > 4167026334 > The rate of the messages per hour is > 50.000 Those are not consistent. Four billion messages per day would be around 50,000 per second. ---
**Author:** ![AlexKonkin](https://avatars.discourse-cdn.com/v4/letter/a/eb9ed0/32.png) [@AlexKonkin](https://discuss.elastic.co/u/AlexKonkin)\ **Post date:** [March 27, 2023, 6:56am UTC](https://discuss.elastic.co/t/logstash-replacement-of-the-coralogix-ruby-output-plugin-with-http-output-plugin-requires-x2-resources/327558/5 "2023-03-27T06:56:31Z")
Hello @leandrojmp Your suggestion was right. After some tests it seems that I found the solution for the issue that I mentioned above. The settings that should optimize the performance of the http output plugin should be done on Logstash level (logstasy.yml) and not on plugin level (the configuration file which uses http output plugin). The 2 settings below helped to keep the same number of pipelines (10) and handle the production load: ```auto pipeline.batch.size: 400 pipeline.batch.delay: 150 ``` Both values by default have the smaller values which were not sifficient to handle the production load: ```auto pipeline.batch.size: 125 pipeline.batch.delay: 50 ``` You can read more about these values by navigating the URL: [https://www.elastic.co/guide/en/logstash/current/logstash-settings-file.html](https://www.elastic.co/guide/en/logstash/current/logstash-settings-file.html) Thank you for your attention to my question. I hope that it could be also helpful to someone who which to optimize the behavior of the http output plugin and logstash application in general. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 24, 2023, 6:59am UTC](https://discuss.elastic.co/t/logstash-replacement-of-the-coralogix-ruby-output-plugin-with-http-output-plugin-requires-x2-resources/327558/7 "2023-04-24T06:59:46Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.