Failed to connect to backoff(async(tcp://localhost:5044)): dial tcp 127.0.0.1:5044: connect: connection refused

[Robin020 (R)]

My setup = filebeat --> logstash --> elasticsearch
The first part (filebeat --> logstash ) doens't work. I got this message in the filebeat log:

INFO    pipeline/output.go:95   Connecting to backoff(async(tcp://localhost:5044))
ERROR   pipeline/output.go:100  Failed to connect to backoff(async(tcp://localhost:5044)): dial tcp 127.0.0.1:5044: connect: connection refused
INFO    pipeline/output.go:93   Attempting to reconnect to backoff(async(tcp://localhost:5044)) with 1 reconnect attempt(s)

When I configure the setup: filebeat --> elasticsearch, it works.

This is my beats-input.conf:

input {
  beats {
    port => 5044
  }
}

And my elasticsearch-output.conf:

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}"
    user => "elastic"
    password => "${ES_PWD}"
  }
  stdout{}
}

This is the output of netstat -plnt:

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:23              0.0.0.0:*               LISTEN      1/systemd
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1890/master
tcp        0      0 0.0.0.0:5601            0.0.0.0:*               LISTEN      13215/node
tcp        0      0 127.0.0.1:9200          0.0.0.0:*               LISTEN      13686/java
tcp        0      0 127.0.0.1:9300          0.0.0.0:*               LISTEN      13686/java
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1368/sshd
tcp6       0      0 :::22                   :::*                    LISTEN      1368/sshd

In the filebeat.yml I have only changed the output to use logstash with port 5044.

I have been troubleshooting for days, but unfortunately I cannot get any further. I hope someone wants to help me.

Robin

[Robin020 (R)]

Ok, I got somehting new.
When I do this: bin/logstash -f /etc/logstash/conf.d/beats-input.conf --config.reload.automatic I see with netstat -plnt that my server is listing on port 5044.
It seems to me that my conf files is not being loaded. I think i missed a step in the process?

Robin

[Sylancer (James)]

Hey Robin

Did you ever figure this out? I'm seeing the same type of message and I think it's causing my Zeek logs to not be shipped at all. I don't see a 5044 port listening and I'm getting those async errors.

[Robin020 (R)]

Hi James,

Yes I have.
When I create a new logstash pipeline config, I do the following steps:

1. Stop logstash services
2. Go to /usr/share/logstash and run bin/logstash --config.test_and_exit -f <path to pipeline config> (this is to check if my config is correct).
3. Stay in the same path and run: bin/logstash -f <path to pipeline config> --config.reload.automatic (when you see the script is running without error, you do CTRL-C)
4. Now you start logstash and your config should loaded (you can check with netstat -plnt if your server is listing on port 5044).

Hope this will help you.

[Sylancer (James)]

I'm happy to say that yes, you did indeed help me. Thanks for taking the time out to not only reply but write easy to read steps and commands. It really helps out a n00b. And now, mostly for my own benefit, here's all the problems I was having and how I (kinda) resolved them.

After reading your post I tried the command in step 2 but I kept on receiving this error:
[ERROR] 2020-01-29 03:24:53.769 [main] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (GemfileNotFound) /usr/share/logstash/Gemfile not found

I couldn't figure this one out for the life of me so going for a hail mary, I did an: apt remove logstash && apt autoremove
Then installed Logstash again via: apt install logstash

I also had 3 different files in /etc/logstash/conf.d. One for input, one for filter and one for output. I consolodated all of these and made one file: 01-BeatsConfig.conf
I tried the command in step 2 and got an error on a line of my 01-BeatsConfig.conf. I had "client_inactivity_timeout" there. So I removed that line. But even so Logstash was complaining about not being able to find the logstash.yml file. I saw that I could set it with the following:

bin/logstash --config.test_and_exit --path.settings /etc/logstash -f /etc/logstash/conf.d/01-BeatsConfig.conf

There's a couple more errors I battled (which I can't remember right now) but after the command above, and the command in step 3, I was finally able to get Logstash listening on 5044. What an ordeal! Thanks again for your help.

[Robin020 (R)]

No problem, I am glad I could help you.