# Troubleshooting too many "Beats input: the pipeline is blocked" messages
**URL:** https://discuss.elastic.co/t/troubleshooting-too-many-beats-input-the-pipeline-is-blocked-messages/83613
**Category:** Logstash
**Created:** [April 25, 2017, 7:03pm UTC](https://discuss.elastic.co/t/troubleshooting-too-many-beats-input-the-pipeline-is-blocked-messages/83613 "2017-04-25T19:03:07Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@elkwil](https://discuss.elastic.co/u/elkwil)
#### Post date: [April 25, 2017, 7:03pm UTC](https://discuss.elastic.co/t/troubleshooting-too-many-beats-input-the-pipeline-is-blocked-messages/83613/1 "2017-04-25T19:03:07Z")
Hi, I have a 4 vCPU, 32GB ram systems with logstash running. I am trying to find my bottleneck and am new to ELK.
For the most part, things work OK, but over the day the logs get about 10-20 minutes behind real time. So I started digging into the logs of the components today.
On a typical day we get about the same amount of events sent from www servers (apache), and looking today from the last 5 hours I have 24,577messages :
`:message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", reconnect_backoff_sleep=>0.5, :level=>:warn}`
About 3.2 million records in that time frame (5 hours)as well have been ingested for today, typical 24hours is 12million events.
`Filebeat is 1.3, logstash is 2.2.4`
During peak load, I have CPU load of around 3.1, networking is 2% utilized, and ram is at 20GB out of 32GB, disk IO is minimal to, and on a SAN.
How can I find out where the bottleneck is here:
My filebeat config:
filebeat:
prospectors:
-
paths:
- /var/log/httpd/_/access\_log._
input\_type: log
exclude\_files: [".gz$"]
document\_type: apache
registry\_file: /var/lib/filebeat/registry
output:
logstash:
hosts: ["elk1-01:5044"]
bulk\_max\_size: 1024
tls:
certificate\_authorities: ["/etc/certs/logstash.crt"]
shipper:
logging:
files:
rotateeverybytes: 10485760 # = 10MB
Logstash:
```
input {
beats {
port => 5044
ssl => true
ssl_certificate => "/etc/pki/logstash.crt"
ssl_key => "/etc/pki/tls/logstash.key"
}
}
filter {
if [type] == "apache" {
grok {
match => ["message", "%{IP:client_ip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:apache_timestamp}\] \"%{WORD:method} /%{NOTSPACE:request_page} HTTP/%{NUMBER:http_version}\" %{NUMBER:server_response} %{NUMBER:bytes} \"%{GREEDYDATA:referer}\" \"%{GREEDYDATA:user_agent}\" %{NUMBER:received} %{NUMBER:sent} \**%{NUMBER:duration_seconds}/%{NUMBER:duration_micro}\*\*" ]
}
date {
match => ["apache_timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
locale => en
}
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
sniffing => true
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
```
Thank You.
---
### Author:  [@elkwil](https://discuss.elastic.co/u/elkwil)
#### Post date: [April 26, 2017, 3:50pm UTC](https://discuss.elastic.co/t/troubleshooting-too-many-beats-input-the-pipeline-is-blocked-messages/83613/2 "2017-04-26T15:50:18Z")
{:timestamp=\>"2017-04-26T15:46:20.636000+0000", :message=\>"CircuitBreaker::rescuing exceptions", :name=\>"Beats input", :exception=\>LogStash::Inputs::Beats::InsertingToQueueTakeTooLong, :level=\>:warn}
I think this is my problem, how to increase the speed of this
---
### Author:  [@elkwil](https://discuss.elastic.co/u/elkwil)
#### Post date: [April 27, 2017, 3:04pm UTC](https://discuss.elastic.co/t/troubleshooting-too-many-beats-input-the-pipeline-is-blocked-messages/83613/3 "2017-04-27T15:04:47Z")
I tried the -w option, but I am, not sure what the default number is for 2.2 logstash. I googled but could not find it.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [May 25, 2017, 3:17pm UTC](https://discuss.elastic.co/t/troubleshooting-too-many-beats-input-the-pipeline-is-blocked-messages/83613/4 "2017-05-25T15:17:03Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.