Troubleshooting too many "Beats input: the pipeline is blocked" messages


#1

Hi, I have a 4 vCPU, 32GB ram systems with logstash running. I am trying to find my bottleneck and am new to ELK.

For the most part, things work OK, but over the day the logs get about 10-20 minutes behind real time. So I started digging into the logs of the components today.

On a typical day we get about the same amount of events sent from www servers (apache), and looking today from the last 5 hours I have 24,577messages :

:message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", reconnect_backoff_sleep=>0.5, :level=>:warn}

About 3.2 million records in that time frame (5 hours)as well have been ingested for today, typical 24hours is 12million events.

Filebeat is 1.3, logstash is 2.2.4

During peak load, I have CPU load of around 3.1, networking is 2% utilized, and ram is at 20GB out of 32GB, disk IO is minimal to, and on a SAN.
How can I find out where the bottleneck is here:

My filebeat config:
filebeat:
prospectors:
-
paths:
- /var/log/httpd//access_log.
input_type: log
exclude_files: [".gz$"]
document_type: apache
registry_file: /var/lib/filebeat/registry
output:
logstash:
hosts: ["elk1-01:5044"]
bulk_max_size: 1024
tls:
certificate_authorities: ["/etc/certs/logstash.crt"]
shipper:
logging:
files:
rotateeverybytes: 10485760 # = 10MB

Logstash:

input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => "/etc/pki/logstash.crt"
    ssl_key => "/etc/pki/tls/logstash.key"
  }
}

filter {
  if [type] == "apache" {
    grok {
	match => [ "message", "%{IP:client_ip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:apache_timestamp}\] \"%{WORD:method} /%{NOTSPACE:request_page} HTTP/%{NUMBER:http_version}\" %{NUMBER:server_response} %{NUMBER:bytes} \"%{GREEDYDATA:referer}\" \"%{GREEDYDATA:user_agent}\" %{NUMBER:received} %{NUMBER:sent} \**%{NUMBER:duration_seconds}/%{NUMBER:duration_micro}\*\*" ]
    }
    date {
  match => [ "apache_timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
   locale => en
    }
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    sniffing => true
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}   

Thank You.


#2

{:timestamp=>"2017-04-26T15:46:20.636000+0000", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Beats input", :exception=>LogStash::Inputs::Beats::InsertingToQueueTakeTooLong, :level=>:warn}

I think this is my problem, how to increase the speed of this


#3

I tried the -w option, but I am, not sure what the default number is for 2.2 logstash. I googled but could not find it.


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.