Logstash/logstash forwarder connection issue

Elastic Stack Logstash
1

Hello, I am trying to send records using logstash forwarder to logstash instance(dedicated server), then from logstash to elasticsearch(another dedicated server) but got this error from logstash:

Lumberjack input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover. {:exception=>LogStash::CircuitBreaker::HalfOpenBreaker, :level=>:warn}
Lumberjack input: the pipeline is blocked, temporary refusing new connection. {:level=>:warn}
CircuitBreaker::Open {:name=>"Lumberjack input", :level=>:warn}
Lumberjack input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover. {:exception=>LogStash::CircuitBreaker::OpenBreaker, :level=>:warn}
Lumberjack input: the pipeline is blocked, temporary refusing new connection. {:level=>:warn}
Lumberjack input: the pipeline is blocked, temporary refusing new connection. {:level=>:warn}
...
...

and this error from logstash-forwarder:
Read error looking for akc:EOF

and it keeps trying to reconnect, and once connected, it's back to the error again

I tested them individually, I was able to send records from logstash-forwarder to logstash and print them to the terminal, I was also able to parse logs with logstash and send them to elasticsearch. But when I connect them together, aka, logstash-forwarder-> logstash -> elasticsearch. I have the situation above? Any suggestions will be appreciated. Thank you!

2

Providing your configs would be useful.

Please make sure you format them with the pre-formatted option!

3

forwarder config:
{ "network": { "servers": [ "server_name:8010" ], "ssl ca": "lumberjack.crt", "timeout": 30 }, "files": [ { "paths": [ "C:/Users/DAVIDL/Desktop/logs/*" ], "fields": { "type": "log" } } ] }

logstash input:
lumberjack { port => 8010 # flush_size => 512 ssl_certificate => "/home/davidl/logstash-2.1.1/ssl/lumberjack.crt" ssl_key => "/home/davidl/logstash-2.1.1/ssl/lumberjack.key" }
logstash output:
lumberjack { hosts => "Another-server" port => 5043 ssl_certificate => "/home/logstash-2.1.1/ssl/logstash-forwarder.crt" codec => "json" }

Thank you for the help!

4

If your Logstash instance is to send logs to Elasticsearch, why does the output contain the lumberjack output plugin?

5

Sorry, I should have made this more clear, the logstash instance is actually sending the logs to another logstash instance before eventually get to elasticsearch, this it the infrastructure set up by the logging team at our company, they want to make sure everything is formatted well in Json before their logstash gets the records. My job is to make sure our logs get parsed and well formatted before sending to them, and I encountered the above in the process.

6

Fixed by changing to Filebeat, I would recommend whoever's having logstash forwarder issues to switch to filebeat first and see if it's resolved.