I have filebeat installed on all clients which send log files to my central logging server.
On the central server I have kibanna, logstash and filebeat installed. The other day I ran out of space. This server is a vm so I increased the disk space, memory and cpu on the vm. I rebooted the server and restarted the filebeat system service on the clients. Now it looks like my server is not receiving any data from the clients even though the fliebeat service is running. Any help fixing this would be helpful.
when i start the service on a client for filebeat i get a timeout
usr/bin/filebeat[17506]: transport.go:125: SSL client failed to connect with: read tcp 192.168.14.112:48368...timeout
Sep 17 11:59:46 /usr/bin/filebeat[17506]: transport.go:125: SSL client failed to connect with: read tcp 192.168.14.112:48382...timeout
Sep 17 12:04:08 /usr/bin/filebeat[17506]: transport.go:125: SSL client failed to connect with: read tcp 192.168.14.112:48388...timeout
Sep 17 12:14:45 /usr/bin/filebeat[17506]: transport.go:125: SSL client failed to connect with: read tcp 192.168.14.112:48414...timeout
Hi @UnNis
You mention that on your central server you have Kibana, Logstash and Filebeat, but filebeat is not needed in a central server to collect the logs from other servers, and you would need also at least an Elasticsearch instance, could you explain your architecture with more detail? is Elasticsearch also on this central server?
sorry thats right filebeat is not on the server. But on the client system. The logging server has kibana, logstash and elasticsearch installed. Its on a centos 7 system.
On the client end I am getting this error:
transport.go:125: SSL client failed to connect with: dial tcp 192.168.1.24:5044: getsockopt: connection refused
I can telnet to the server from the client on port 5044
The errors you posted look like some connectivity error, it looks strange that in the first post the problem was timeout and later it was connection refused.
Can you check that logstash is running and it doesn't report any error in its logs?
Here is the entries from the logstash logs
{:timestamp=>"2018-09-17T03:23:04.618000+0100", :message=>"Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::Inputs::BeatsSupport::CircuitBreaker::HalfOpenBreaker, :level=>:warn}
{:timestamp=>"2018-09-17T03:23:23.426000+0100", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2018-09-17T03:23:23.426000+0100", :message=>"CircuitBreaker::Open", :name=>"Beats input", :level=>:warn}
{:timestamp=>"2018-09-17T03:23:23.426000+0100", :message=>"Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::Inputs::BeatsSupport::CircuitBreaker::OpenBreaker, :level=>:warn}
{:timestamp=>"2018-09-17T03:23:23.926000+0100", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2018-09-17T03:23:24.426000+0100", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2018-09-17T03:23:24.927000+0100", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
above is from the logstash log.
{:timestamp=>"2018-09-17T18:42:21.187000+0100", :message=>"Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::Inputs::BeatsSupport::CircuitBreaker::HalfOpenBreaker, :level=>:warn}
{:timestamp=>"2018-09-17T18:42:21.248000+0100", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Beats input", :exception=>LogStash::Inputs::Beats::InsertingToQueueTakeTooLong, :level=>:warn}
{:timestamp=>"2018-09-17T18:42:21.250000+0100", :message=>"Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::Inputs::BeatsSupport::CircuitBreaker::HalfOpenBreaker, :level=>:warn}
{:timestamp=>"2018-09-17T18:42:21.356000+0100", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2018-09-17T18:42:21.857000+0100", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2018-09-17T18:42:22.357000+0100", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2018-09-17T18:42:22.858000+0100", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2018-09-17T18:42:23.358000+0100", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2018-09-17T18:42:23.859000+0100", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2018-09-17T18:42:24.359000+0100", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2018-09-17T18:42:24.860000+0100", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2018-09-17T18:42:25.360000+0100", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2018-09-17T18:42:25.861000+0100", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2018-09-17T18:42:26.361000+0100", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2018-09-17T18:42:26.862000+0100", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2018-09-17T18:42:27.362000+0100", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2018-09-17T18:42:27.863000+0100", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2018-09-17T18:42:28.363000+0100", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2018-09-17T18:42:28.864000+0100", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2018-09-17T18:42:29.364000+0100", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2018-09-17T18:42:29.865000+0100", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2018-09-17T18:42:30.366000+0100", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2018-09-17T18:42:30.866000+0100", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2018-09-17T18:42:31.367000+0100", :message=>"Beats input: the pipeline is blocked, }
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.