My setup = filebeat --> logstash --> elasticsearch
The first part (filebeat --> logstash ) doens't work. I got this message in the filebeat log:
INFO pipeline/output.go:95 Connecting to backoff(async(tcp://localhost:5044))
ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://localhost:5044)): dial tcp 127.0.0.1:5044: connect: connection refused
INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://localhost:5044)) with 1 reconnect attempt(s)
When I configure the setup: filebeat --> elasticsearch, it works.
This is my beats-input.conf:
input {
beats {
port => 5044
}
}
And my elasticsearch-output.conf:
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "%{[@metadata][beat]}-%{[@metadata][version]}"
user => "elastic"
password => "${ES_PWD}"
}
stdout{}
}
Ok, I got somehting new.
When I do this: bin/logstash -f /etc/logstash/conf.d/beats-input.conf --config.reload.automatic I see with netstat -plnt that my server is listing on port 5044.
It seems to me that my conf files is not being loaded. I think i missed a step in the process?
Did you ever figure this out? I'm seeing the same type of message and I think it's causing my Zeek logs to not be shipped at all. I don't see a 5044 port listening and I'm getting those async errors.
Yes I have.
When I create a new logstash pipeline config, I do the following steps:
Stop logstash services
Go to /usr/share/logstash and run bin/logstash --config.test_and_exit -f <path to pipeline config> (this is to check if my config is correct).
Stay in the same path and run: bin/logstash -f <path to pipeline config> --config.reload.automatic (when you see the script is running without error, you do CTRL-C)
Now you start logstash and your config should loaded (you can check with netstat -plnt if your server is listing on port 5044).
I'm happy to say that yes, you did indeed help me. Thanks for taking the time out to not only reply but write easy to read steps and commands. It really helps out a n00b. And now, mostly for my own benefit, here's all the problems I was having and how I (kinda) resolved them.
After reading your post I tried the command in step 2 but I kept on receiving this error:
[ERROR] 2020-01-29 03:24:53.769 [main] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (GemfileNotFound) /usr/share/logstash/Gemfile not found
I couldn't figure this one out for the life of me so going for a hail mary, I did an: apt remove logstash && apt autoremove
Then installed Logstash again via: apt install logstash
I also had 3 different files in /etc/logstash/conf.d. One for input, one for filter and one for output. I consolodated all of these and made one file: 01-BeatsConfig.conf
I tried the command in step 2 and got an error on a line of my 01-BeatsConfig.conf. I had "client_inactivity_timeout" there. So I removed that line. But even so Logstash was complaining about not being able to find the logstash.yml file. I saw that I could set it with the following:
There's a couple more errors I battled (which I can't remember right now) but after the command above, and the command in step 3, I was finally able to get Logstash listening on 5044. What an ordeal! Thanks again for your help.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.