Hi, I have a 4 vCPU, 32GB ram systems with logstash running. I am trying to find my bottleneck and am new to ELK.
For the most part, things work OK, but over the day the logs get about 10-20 minutes behind real time. So I started digging into the logs of the components today.
On a typical day we get about the same amount of events sent from www servers (apache), and looking today from the last 5 hours I have 24,577messages :
:message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", reconnect_backoff_sleep=>0.5, :level=>:warn}
About 3.2 million records in that time frame (5 hours)as well have been ingested for today, typical 24hours is 12million events.
Filebeat is 1.3, logstash is 2.2.4
During peak load, I have CPU load of around 3.1, networking is 2% utilized, and ram is at 20GB out of 32GB, disk IO is minimal to, and on a SAN.
How can I find out where the bottleneck is here:
My filebeat config:
filebeat:
prospectors:
-
paths:
- /var/log/httpd//access_log.
input_type: log
exclude_files: [".gz$"]
document_type: apache
registry_file: /var/lib/filebeat/registry
output:
logstash:
hosts: ["elk1-01:5044"]
bulk_max_size: 1024
tls:
certificate_authorities: ["/etc/certs/logstash.crt"]
shipper:
logging:
files:
rotateeverybytes: 10485760 # = 10MB
Logstash:
input {
beats {
port => 5044
ssl => true
ssl_certificate => "/etc/pki/logstash.crt"
ssl_key => "/etc/pki/tls/logstash.key"
}
}
filter {
if [type] == "apache" {
grok {
match => [ "message", "%{IP:client_ip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:apache_timestamp}\] \"%{WORD:method} /%{NOTSPACE:request_page} HTTP/%{NUMBER:http_version}\" %{NUMBER:server_response} %{NUMBER:bytes} \"%{GREEDYDATA:referer}\" \"%{GREEDYDATA:user_agent}\" %{NUMBER:received} %{NUMBER:sent} \**%{NUMBER:duration_seconds}/%{NUMBER:duration_micro}\*\*" ]
}
date {
match => [ "apache_timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
locale => en
}
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
sniffing => true
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
Thank You.