Logstash not starting properly

motts · February 16, 2018, 6:54pm

Hello there,

I am trying to start LogStash 5.6 with /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d but when I do, I get several of these errors

[ERROR] 2018-02-16 13:50:24.556 [[main]<tcp] pipeline - A plugin had an unrecoverable error. Will restart this plugin.
  Plugin: <LogStash::Inputs::Tcp type=>"firewall", port=>5009, id=>"3e7ce27249ce9605710e23e87684adbe6de42a6f-9", enable_metric=>true, codec=><LogStash::Codecs::Line id=>"line_e1350e07-231b-48d5-9498-8360bff910d4", enable_metric=>true, charset=>"UTF-8", delimiter=>"\n">, host=>"0.0.0.0", data_timeout=>-1, mode=>"server", proxy_protocol=>false, ssl_enable=>false, ssl_verify=>true, ssl_key_passphrase=><password>>
  Error: Address already in use

I am not sure what is going on here.

Thanks

magnusbaeck · February 16, 2018, 8:28pm

There's already a program listing on port 5009. That program could be Logstash itself. What files do you have in /etc/logstash/conf.d? Keep in mind that Logstash 5 reads all files in that directory.

motts · February 16, 2018, 8:45pm

I have NxLog configured to send stuff on that port to the logstash server, but it only sends stuff if something on the other end is configured to receive it, so I hope in this case it's logstash. I found online where other people put other files in the conf.d directory, but the only ones I have in there are the input, filters, and output files though there is a filter in there. That error posted above appears for all the .conf files in that directory.

Just ran /etc/init.d/logstash.rpmsave configtest and also with -f /etc/logstash and -f /etc/logstash/conf.d but they all produce the same error, which I found to be normal

WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults ERROR: Unrecognised option '--configtest'

I try to run /usr/share/logstash/bin/logstash -f /etc/logstash.conf again, but I get these errors to display. There are several errors like this for each [type] in my logstash.conf files

[root@localhost ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/ WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults

Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console

[ERROR] 2018-02-16 15:38:27.306 [[main]<tcp] pipeline - A plugin had an unrecoverable error. Will restart this plugin. Plugin: <LogStash::Inputs::Tcp type=>"ftp", port=>5007, id=>"3e7ce27249ce9605710e23e87684adbe6de42a6f-7", enable_metric=>true, codec=><LogStash::Codecs::Line id=>"line_e0ddcd42-ab97-47e2-b311-6a19d1cfb611", enable_metric=>true, charset=>"UTF-8", delimiter=>"\n">, host=>"0.0.0.0", data_timeout=>-1, mode=>"server", proxy_protocol=>false, ssl_enable=>false, ssl_verify=>true, ssl_key_passphrase=><password>> Error: Address already in use

[ERROR] 2018-02-16 15:38:27.309 [[main]<tcp] pipeline - A plugin had an unrecoverable error. Will restart this plugin. Plugin: <LogStash::Inputs::Tcp type=>"iis", port=>5001, codec=><LogStash::Codecs::Line id=>"line_69db6bfb-16b0-4607-8a23-6ee58d709f59", enable_metric=>true, charset=>"UTF-8", delimiter=>"\n">, id=>"3e7ce27249ce9605710e23e87684adbe6de42a6f-2", enable_metric=>true, host=>"0.0.0.0", data_timeout=>-1, mode=>"server", proxy_protocol=>false, ssl_enable=>false, ssl_verify=>true, ssl_key_passphrase=><password>> Error: Address already in use

This is in the logstash-stderr log
/usr/share/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-5.0.4/lib/elasticsearch/transport/transport/connections/connection.rb:117 warning: out of Float range

and this in the logstash-stdout log
2018-02-16 13:40:20,216 main ERROR Unable to create file /var/log/logstash/logstash-plain.log java.io.IOException: No such file or directory

but that log file should be created if it doesn't exist

motts · February 16, 2018, 9:17pm

Also, if this helps, I found some articles that said to edit the logstash.rpmsave file to replace program=/opt/... with program=/usr/share/... and to comment out the line below that args="agent -f ${LS_CONF_DIR} -l ${LS_LOG_FILE} ${LS_OPTS}"

I also added path.settings: /etc/logstash to the logstash.yml file

motts · February 16, 2018, 9:59pm

LogStash still seems to be working, but there are errors everywhere like you see above. Even running /etc/init.d/logstash.rpmsave status logstash says logstash is not running, though it is. This was so much simpler with service logstash start lol

magnusbaeck · February 19, 2018, 7:09am

Start by fixing the "address already in use" problem. As I said, when Logstash starts up there's already some process listening on that address and port. Dig into that, for example using netstat.

motts · February 19, 2018, 3:19pm

Ok. I rebooted the server and everything seems fine. There are no longer errors in any of the log files; however, when I check on /etc/init.d/logstash.rpmsave status it says logstash is not running though I am seeing logs in kibana and initctl status logstash says logstash start/running, process 1044 . Should I just ignore logstash.rpmsave status? To my understanding, that is how I should restart logstash in the event I make a change to one of the conf files. Should I use initctl restart logstash instead?

magnusbaeck · February 19, 2018, 3:52pm

It sounds like the .rpmsave file is out of date or pointing to the wrong files, but I'm not RedHat guy.

motts · February 19, 2018, 4:02pm

No worries. I'll post up my config and maybe I can get someone to chime in.

I have only modified 2 lines in my logstash.rpmsave:

program=/usr/share/logstash/bin/logstash
#args="agent -f ${LS_CONF_DIR} -l ${LS_LOG_FILE} ${LS_OPTS}"

I changed the program line from /opt to /usr/share and I commented out the args line. Everything else is default.

When I attempt to start logstash with /etc/init.d/logstash.rpmsave start these errors appear in logstash.err

WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
ERROR: No configuration file was specified. Perhaps you forgot to provide the '-f yourlogstash.conf' flag?

and this in logstash.stdout

Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console

So I try this instead

/etc/init.d/logstash.rpmsave start -f /etc/logstash/conf.d --path.settings /etc/logstash but I get the same errors showing up. There is also a log entry saying ERROR: Unknown command 'agent' and I found an article saying to just comment out that line in the rpmsave.

magnusbaeck · February 19, 2018, 6:55pm

Where does this file come from and why do you have to make changes to it? Why are you commenting the args assignment? If you have an RPM-based system just install the RPM package and you shouldn't have to touch any init scripts.

/etc/init.d/logstash.rpmsave start -f /etc/logstash/conf.d --path.settings /etc/logstash

You can't supply program arguments in this way.

motts · February 19, 2018, 7:14pm

The file came from updating logstash. When I have the args= line uncommented, I get errors stating ERROR: Unknown command 'agent'. I did some digging and found a couple articles saying to comment the line out of the rpmsave file.

Is this not the repo for the rpm package? I used this exact one for ES and Kibana and have no issues with either one running.

[logstash-5.x]
name=Logstash repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

From here, I would do yum update logstash

magnusbaeck · February 19, 2018, 9:23pm

Judging by https://serverfault.com/a/48819 your rpmsave file is a copy of your old init script that you modified, and instead of overwriting the file (or deleting it?) rpm saves it for you. That would explain why it contains the "agent" reference and doesn't work.

So, there shouldn't be a /etc/init.d/logstash.rpmsave in a well-maintained system and unless you know exactly what you're doing you shouldn't modify it or attempt to use it for starting a service.

motts · February 19, 2018, 9:27pm

Yeah I had no idea what I was doing. That is why we have labs right? haha

I think I got a streamlined process now. That does explain why it was showing logstash was already running. During the upgrade process, the logstash service never stopped and logs could still be sent. I would try to start it the new way and get the errors so I got confused.

My mistake here. Thank you for clearing that up for me.

system · March 19, 2018, 9:28pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.