Logstash Error: Beats input: the pipeline is blocked, temporary refusing new connection

Elastic Stack Logstash
1

After pointing 8 or so filebeat agents to my logstash server, I'm getting this message in the logstash log:
Beats input: the pipeline is blocked, temporary refusing new connection.

What is the cause of this error?

At first I assumed it was because I don't have a redis queue in the stack.. But from everything I've read, I'm very confused as to if I should use redis between Filebeat or not.

Is this error because I need a Redis message queue, or something else?

Pasted below are my filebeat inputs and ES outputs:

input {
  beats {
    port => 5044
    type => "logs"
    ssl => true
    ssl_certificate => "/etc/pki/tls/certs/somecert.crt"
    ssl_key => "/etc/pki/tls/private/somecert.key"
    congestion_threshold => "40"
  }
}

output {
  elasticsearch {
    hosts => ["my_es_client_node_lb:9200"]
    user => logstash
    password => my_pw
  }

  stdout { codec => rubydebug }
}
2

FYI I've resolved my issue it looks like.. By condensing all of my pipleline config into one file, I seemed to have silienced the error I was getting before..

I'm indexing up to 900 events per second, from filebeat to logstash directly, and logstash isn't complaining at all.

1 Like
3

Hi @Mike_Wurtz could you please share you're config I'm new to ELK don't know what to do