# Slowness in logstash throughput while reading from topbeat : how to debug **URL:** \ **Category:** Logstash\ **Created:** [January 13, 2016, 11:20pm UTC](https://discuss.elastic.co/t/slowness-in-logstash-throughput-while-reading-from-topbeat-how-to-debug/39165 "2016-01-13T23:20:26Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![biswajit86](https://avatars.discourse-cdn.com/v4/letter/b/f17d59/32.png) [@biswajit86](https://discuss.elastic.co/u/biswajit86)\ **Post date:** [January 13, 2016, 11:20pm UTC](https://discuss.elastic.co/t/slowness-in-logstash-throughput-while-reading-from-topbeat-how-to-debug/39165/1 "2016-01-13T23:20:26Z")
Hi, I am trying to use logstash to consume messages from topbeat. I setup a topbeat-\>logstash-\>elasticsearch flow and I can see the data in kibana. However, I keep on seeing these messages in logstash logs 1. `Beats input: the pipeline is blocked, temporary refusing new connection. {:level=>:warn} CircuitBreaker::Close {:name=>"Beats input", :level=>:warn}` 2.`CircuitBreaker::rescuing exceptions {:name=>"Beats input", :exception=>LogStash::SizedQueueTimeout::TimeoutError, :level=>:warn}` 2. `Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover. {:exception=>LogStash::CircuitBreaker::HalfOpenBreaker, :level=>:warn}` I have a few questions regarding these 1. How can I figure out where there is a delay in throughput , is it - the logstash input worker , - the logstash output worker , - elastic indexing throughput 2. Is there a way to monitor the indexing performance of elasticsearch cluster /node ? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 14, 2016, 6:21am UTC](https://discuss.elastic.co/t/slowness-in-logstash-throughput-while-reading-from-topbeat-how-to-debug/39165/2 "2016-01-14T06:21:57Z")
You can use [Marvel](https://www.elastic.co/guide/en/marvel/current/index.html) for monitoring your cluster. For Logstash though you'd want to look at the [metrics](https://www.elastic.co/guide/en/logstash/current/plugins-filters-metrics.html) filter. We're working on exposing more monitoring functionality with upcoming versions of LS. ---
**Author:** ![biswajit86](https://avatars.discourse-cdn.com/v4/letter/b/f17d59/32.png) [@biswajit86](https://discuss.elastic.co/u/biswajit86)\ **Post date:** [January 14, 2016, 5:08pm UTC](https://discuss.elastic.co/t/slowness-in-logstash-throughput-while-reading-from-topbeat-how-to-debug/39165/3 "2016-01-14T17:08:24Z")
Could you please tell me what metric points can i see in marvel to figure out any issues. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 14, 2016, 9:48pm UTC](https://discuss.elastic.co/t/slowness-in-logstash-throughput-while-reading-from-topbeat-how-to-debug/39165/4 "2016-01-14T21:48:07Z")
You can look at things like indexing throughput to see if it drops when you get this log entry. ---
**Author:** ![biswajit86](https://avatars.discourse-cdn.com/v4/letter/b/f17d59/32.png) [@biswajit86](https://discuss.elastic.co/u/biswajit86)\ **Post date:** [January 15, 2016, 9:08pm UTC](https://discuss.elastic.co/t/slowness-in-logstash-throughput-while-reading-from-topbeat-how-to-debug/39165/5 "2016-01-15T21:08:52Z")
Here is where I am stuck. I am unable to find the point where there is a congestion in the stack when i see the below message ``` CircuitBreaker::rescuing exceptions {:name=>"Beats input", :exception=>LogStash::SizedQueueTimeout::TimeoutError, :level=>:warn} Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover. {:exception=>LogStash::CircuitBreaker::HalfOpenBreaker, :level=>:warn} ``` I am unable to diagnose whether the elastic instance is slow or if the logstash instance is slow. I do not see any peritnent information in either elastic/logstash ) logs which explains when and how the pipeline is determined to be in a blocked state. Unless i can determine that, i am not sure if I need more logstash instances, more elastic instances, dedicated master node for elastic or a queueing solution. How do I find out/deduce this piece of information ---
**Author:** ![Omar\_Al\_Zabir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/omar_al_zabir/32/5943_2.png) [@Omar\_Al\_Zabir](https://discuss.elastic.co/u/Omar_Al_Zabir)\ **Post date:** [February 22, 2016, 2:44pm UTC](https://discuss.elastic.co/t/slowness-in-logstash-throughput-while-reading-from-topbeat-how-to-debug/39165/6 "2016-02-22T14:44:43Z")
Same problem here. I keep getting this problem throughout the day. Can't figure out why it is causing the problem every now and then. We definitely need better logging and monitoring capability in LS. ---
**Author:** ![Omar\_Al\_Zabir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/omar_al_zabir/32/5943_2.png) [@Omar\_Al\_Zabir](https://discuss.elastic.co/u/Omar_Al_Zabir)\ **Post date:** [February 23, 2016, 11:18am UTC](https://discuss.elastic.co/t/slowness-in-logstash-throughput-while-reading-from-topbeat-how-to-debug/39165/7 "2016-02-23T11:18:08Z")
I have a pretty modest filter and \< 100 TPS load on a 4 CPU server, still I get circuitbreaker tripped. I can see CPU usage is 200%+ by the logstash java process. ```auto input { beats { port => 5045 type => 'iis' } } # First filter filter { #ignore log comments if [message] =~ "^#" { drop {} } grok { patterns_dir => "./patterns" match => ["message", "%{TIMESTAMP_ISO8601:timestamp} %{IPORHOST:serverip} %{WORD:verb} %{PATH:request} %{NOTSPACE:querystring} %{NUMBER:port} %{NOTSPACE:auth} %{IPORHOST:clientip} %{NOTSPACE:agent} %{NUMBER:response} %{NUMBER:sub_response} %{NUMBER:sc_status} %{NUMBER:responsetime}"] } date { match => ["timestamp", "yyyy-MM-dd HH:mm:ss"] locale => "en" } } # Second filter filter { if "_grokparsefailure" in [tags] { } else { # on success remove the message field to save space mutate { remove_field => ["message", "timestamp"] } } } output { if [system] { elasticsearch { hosts => ["10.35.132.143:9200","10.35.132.142:9200","10.35.76.37:9200"] index => "logstash-%{system}-%{group}-%{+YYYY.MM.dd}" template => "./conf/apache-mapping.json" template_name => "logstash" document_type => "%{type}" template_overwrite => true manage_template => true } } else { elasticsearch { hosts => ["10.35.132.143:9200","10.35.132.142:9200","10.35.76.37:9200"] index => "junk" document_type => "%{type}" } } #stdout { codec => rubydebug } } ``` ---
**Author:** ![remco](https://avatars.discourse-cdn.com/v4/letter/r/9d8465/32.png) [@remco](https://discuss.elastic.co/u/remco)\ **Post date:** [April 6, 2016, 5:53pm UTC](https://discuss.elastic.co/t/slowness-in-logstash-throughput-while-reading-from-topbeat-how-to-debug/39165/8 "2016-04-06T17:53:07Z")
Hi, I'm having simular problems (error messages). Just curious if you have already solved the issue? For me it unsure if it a setting problem or grok being busy (multiple core 100% cpu) which might cause connection/pipeline problems. ---
**Author:** ![Omar\_Al\_Zabir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/omar_al_zabir/32/5943_2.png) [@Omar\_Al\_Zabir](https://discuss.elastic.co/u/Omar_Al_Zabir)\ **Post date:** [April 14, 2016, 10:23pm UTC](https://discuss.elastic.co/t/slowness-in-logstash-throughput-while-reading-from-topbeat-how-to-debug/39165/9 "2016-04-14T22:23:03Z")
In my case it was the grok pattern. I had to make the pattern simpler to make it work. ---
**Author:** ![ally](https://avatars.discourse-cdn.com/v4/letter/a/8edcca/32.png) [@ally](https://discuss.elastic.co/u/ally)\ **Post date:** [April 25, 2016, 3:46pm UTC](https://discuss.elastic.co/t/slowness-in-logstash-throughput-while-reading-from-topbeat-how-to-debug/39165/10 "2016-04-25T15:46:11Z")
Hello, I have the same problem with similar configuration of logstash (just a grok and date in the filter). How did you split it? I have also tried but with not success. Thanks! ---
**Author:** ![remco](https://avatars.discourse-cdn.com/v4/letter/r/9d8465/32.png) [@remco](https://discuss.elastic.co/u/remco)\ **Post date:** [April 25, 2016, 7:26pm UTC](https://discuss.elastic.co/t/slowness-in-logstash-throughput-while-reading-from-topbeat-how-to-debug/39165/11 "2016-04-25T19:26:04Z")
For me it is not solved (i changed back to logstash instances on ever machine). I changed the grok pattern which made logstash processes the message somewhat faster. ---
**Author:** ![ally](https://avatars.discourse-cdn.com/v4/letter/a/8edcca/32.png) [@ally](https://discuss.elastic.co/u/ally)\ **Post date:** [April 25, 2016, 8:50pm UTC](https://discuss.elastic.co/t/slowness-in-logstash-throughput-while-reading-from-topbeat-how-to-debug/39165/12 "2016-04-25T20:50:23Z")
What did you change in the grok? @Omar_Al_Zabir said that he has splitted into several. I have a similar match but I was not able to make it work. ---
**Author:** ![navox19](https://avatars.discourse-cdn.com/v4/letter/n/49beb7/32.png) [@navox19](https://discuss.elastic.co/u/navox19)\ **Post date:** [April 28, 2016, 9:52pm UTC](https://discuss.elastic.co/t/slowness-in-logstash-throughput-while-reading-from-topbeat-how-to-debug/39165/13 "2016-04-28T21:52:58Z")
any help ? ---
**Author:** ![ally](https://avatars.discourse-cdn.com/v4/letter/a/8edcca/32.png) [@ally](https://discuss.elastic.co/u/ally)\ **Post date:** [July 13, 2016, 8:56am UTC](https://discuss.elastic.co/t/slowness-in-logstash-throughput-while-reading-from-topbeat-how-to-debug/39165/14 "2016-07-13T08:56:04Z")
Not, I still haven't found the solution. I have split each service into different VM and slow down the amount of messages per second, but this does not really solve the problem. ---
**Author:** ![Omar\_Al\_Zabir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/omar_al_zabir/32/5943_2.png) [@Omar\_Al\_Zabir](https://discuss.elastic.co/u/Omar_Al_Zabir)\ **Post date:** [November 18, 2016, 11:08am UTC](https://discuss.elastic.co/t/slowness-in-logstash-throughput-while-reading-from-topbeat-how-to-debug/39165/15 "2016-11-18T11:08:01Z")
You can test the reg ex you are using on [https://www.regex101.com/](https://www.regex101.com/) and see the time and cycle it takes. If it is over 100 then your regex is too expensive and thus logstash will jam. ---
**Author:** ![seanziee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/seanziee/32/45151_2.png) [@seanziee](https://discuss.elastic.co/u/seanziee)\ **Post date:** [February 27, 2017, 7:25pm UTC](https://discuss.elastic.co/t/slowness-in-logstash-throughput-while-reading-from-topbeat-how-to-debug/39165/16 "2017-02-27T19:25:57Z")
Still nothing? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:28am UTC](https://discuss.elastic.co/t/slowness-in-logstash-throughput-while-reading-from-topbeat-how-to-debug/39165/17 "2017-07-06T04:28:15Z")