Hi.
I have two servers with logstash-forwarder.
First server send logs in a json format.
Second server send logs in a plain format.
On a third server, I have ELK.
I configure Logstash like this :
input {
Get files from logstash forwarders
lumberjack {
port => 5000
type => "logs"
ssl_certificate => "/etc/logstash/myhost.crt"
ssl_key => "/etc/logstash/myhost.key"
codec => json {
charset => "Windows-1252"
}
}
}
filter {
if [type] == "plain" {
grok {
match => { "message" => "aGrokExpression"}
}
}
}
output {
if "_grokparsefailure" in [tags] or "_dateparsefailure" in [tags] {
file {
path => "/var/log/logstash/parsefailure.log"
}
}
elasticsearch {
hosts => "myhost"
manage_template => false
index => "logstash-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
When I run logstash, It works 5 minutes and after that I get this error :
{:timestamp=>"2016-04-28T15:06:44.555000+0200", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Lumberjack input", :exception=>LogStash::SizedQueueTimeout::TimeoutError, :level=>:warn}
{:timestamp=>"2016-04-28T15:06:44.563000+0200", :message=>"Lumberjack input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::CircuitBreaker::HalfOpenBreaker, :level=>:warn}
{:timestamp=>"2016-04-28T15:06:44.721000+0200", :message=>"Lumberjack input: the pipeline is blocked, temporary refusing new connection.", :level=>:warn}
{:timestamp=>"2016-04-28T15:06:44.766000+0200", :message=>"CircuitBreaker::Open", :name=>"Lumberjack input", :level=>:warn}
{:timestamp=>"2016-04-28T15:06:44.767000+0200", :message=>"Lumberjack input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::CircuitBreaker::OpenBreaker, :level=>:warn}
{:timestamp=>"2016-04-28T15:06:45.222000+0200", :message=>"Lumberjack input: the pipeline is blocked, temporary refusing new connection.", :level=>:warn}
I tried to increase the heap_size of Elasticsearch and Logstash to 4Go, but it does not solve the problem.
I also tried to add two lumberjack input (one with json codec and one with plain codec) with no success.
What can I do to solve this ?