Logstash stopped pushing logs to elastic(UPD)

Elastic Stack Logstash
1

Hey friends!
My question about ELK stack. I have remote rails server. It writes two logs - production.log and reports.log. Also I didn't remove ELK's server syslog from my logstash input. So my logstash has to parse local syslog and prod.log and reports.log from remote host. It works until 27th of june O_O. It is not a priority issue. But I have to fix it for better tomorrow;).
I've started this issue 3-4 times and don't have enough patience.
What I've already found:
Filebeat:

/usr/bin/filebeat[24911]: transport.go:125: SSL client failed to connect with: dial tcp x.x.x.x:5044: getsockopt: connection refused

this forum says that it is problem with logstash.
When logstash running:
I can connect with tellnet to this port and ns works too.
But I don't see logs in elastic(kibana)...
Logstash logs says next:

{:timestamp=>"2017-07-14T19:00:08.690000-0400", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Beats input", :exception=>LogStash::Inputs::Beats::InsertingToQueueTakeTooLong, :level=>:warn}
{:timestamp=>"2017-07-14T19:00:08.693000-0400", :message=>"Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::Inputs::BeatsSupport::CircuitBreaker::HalfOpenBreaker, :level=>:warn}
{:timestamp=>"2017-07-14T19:00:08.752000-0400", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Beats input", :exception=>LogStash::Inputs::Beats::InsertingToQueueTakeTooLong, :level=>:warn}
{:timestamp=>"2017-07-14T19:00:08.757000-0400", :message=>"Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::Inputs::BeatsSupport::CircuitBreaker::HalfOpenBreaker, :level=>:warn}

and many many

{:timestamp=>"2017-07-14T19:00:38.821000-0400", :message=>"retrying failed action with response code: 503", :level=>:warn}

I can see syslog data into kibana, but not logs from remote host.

Do you need some extra info? Any configs?

2

Ok. I've remove syslog filter and add output rubydebug. And output looks like logstas gets data from remote:

{
            "message" => "some json data which I can't post",
            ...
}

So output looks correct.
My output file(30-elasticsearch-output.conf) looks:

output {
  stdout { codec => rubydebug }
  elasticsearch {
    hosts => ["localhost:9200"]
    sniffing => true
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

And logstash logs:

{:timestamp=>"2017-07-17T09:19:33.094000-0400", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Beats input", :exception=>LogStash::Inputs::Beats::InsertingToQueueTakeTooLong, :level=>:warn}
{:timestamp=>"2017-07-17T09:19:33.098000-0400", :message=>"Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::Inputs::BeatsSupport::CircuitBreaker::HalfOpenBreaker, :level=>:warn}
{:timestamp=>"2017-07-17T09:19:39.212000-0400", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Beats input", :exception=>LogStash::Inputs::Beats::InsertingToQueueTakeTooLong, :level=>:warn}
{:timestamp=>"2017-07-17T09:19:39.221000-0400", :message=>"Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::Inputs::BeatsSupport::CircuitBreaker::HalfOpenBreaker, :level=>:warn}
{:timestamp=>"2017-07-17T09:19:46.429000-0400", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Beats input", :exception=>LogStash::Inputs::Beats::InsertingToQueueTakeTooLong, :level=>:warn}
{:timestamp=>"2017-07-17T09:19:46.433000-0400", :message=>"Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::Inputs::BeatsSupport::CircuitBreaker::HalfOpenBreaker, :level=>:warn}

in the same time tail -f /var/log/elasticsearch/elasticsearch.log doesn't show new lines. I say "nothing happened".

3

Ok. I've found these answer - Lots of Beats input
It says add congestion_threshold
And I've added. So now logs contains only

retrying failed action with response code: 503

I've found Retrying failed action with response code: 503
ugh:

curl -XGET http://localhost:9200/_cluster/health

returns

{"cluster_name":"elasticsearch","status":"red","timed_out":false,"number_of_nodes":1,"number_of_data_nodes":1...}

status "red" :confused:
Can I fix that without removing all my data and/or indixes ?

4

Can I fix that without removing all my data and/or indixes ?

Maybe, but that's a question for the Elasticsearch group.

5

Thank you. I think it is not a big deal as I've already removed all indices. So my question can be closed. But does it make any sense? Maybe I should remove it(question/topic)?

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.