Logstash 5 not running

Vishal_Sharma1 · October 31, 2016, 1:20pm

hey there,

I am following the official document and seems like it's not working. I am using a freshly installed CentOS 7 system and have Java 1.8 installed. As per the document when i run below
[Foo@Elastic logstash]$ sudo bin/logstash -e 'input { stdin { } } output { stdout {} }'

It just stays like that no output. I have waited for 15 mins now. Any pointers why this is happening?
Thanks
V

Vishal_Sharma1 · October 31, 2016, 1:32pm

Also after an long time i just typed Hello and below is what i have got

INFO: Logstash requires a setting file which is typically located in $LS_HOME/config or /etc/logstash. If you installed Logstash through a package and are starting it manually please specify the location to this settings file by passing in "--path.settings=/path/.."
ERROR: Failed to load settings file from "path.settings". Aborting... path.setting=/usr/share/logstash/config, exception=Errno::ENOENT, message=>No such file or directory

Why's that ?

amit_oren · October 31, 2016, 3:28pm

Well, as the document says - if you don't start logstash via the service, you should pass logstash.yml through the --path.settings i.e.
sudo bin/logstash --path.settings=/etc/logstash/logstash.yml -e 'input { stdin { } } output { stdout {} }'
Assuming you've installed it via the package.

Vishal_Sharma1 · November 1, 2016, 4:28am

Ah really ? but it's not even mention in the official document that i need to include --path.settings=/etc/logstash/logstash.yml. Few things in here worth checking

Why we need to provide --path.settings? it should be a default behavior of logstash to pick up the file from default location.
Why it's not mentioned in the official document?
Even running bin/logstash --path.settings=/etc/logstash/logstash.yml -e 'input { stdin { } } output { stdout {} }'
is failing and giving me below error

INFO: Logstash requires a setting file which is typically located in $LS_HOME/config or /etc/logstash. If you installed Logstash through a package and are starting it manually please specify the location to this settings file by passing in "--path.settings=/path/.."
ERROR: Failed to load settings file from "path.settings". Aborting... path.setting=/etc/logstash/logstash.yml, exception=Errno::ENOENT, message=>No such file or directory - /etc/logstash/logstash.yml/logstash.yml

Regards
V

gabriel_tessier · November 1, 2016, 8:30am

Hi,
I also have a similar problem following the official documentation, installing on an Amzon instance (CentOS 7).

The documentation is correct until "sudo yum install logstash".
After the command chkconfig --add logstash the documentation is not correct cause you need to be root to add it at boot so there's a sudo missing.
Even with sudo this command is not working the logstash bin is not found.

I expect that is the problem so that why Vishal_Sharma1 can't start it as service and try to run the bin.

Somebody can confirm that the yum installation is not working, can it be a rpm problem?
Cause I only copy paste the command and stop after the chkconfig --add logstash.

I checked in the /etc/init.d there's no logstash.

Thanks for any help.

amit_oren · November 1, 2016, 9:06am

Huh, it seems like you actually need to give it the directory, not the file itself (No such file or directory - /etc/logstash/logstash.yml/logstash.yml)
Try changing it to --path.settings=/etc/logstash
It does seem the documentation is not 100% up to date - You can see the changes (and where they say you should specify --path.settings) at the breaking changes page

Vishal_Sharma1 · November 1, 2016, 4:30pm

ok let me clear few things in here

I have installed the logstash from repo.
There is no mention about path.settings in official doc.
A new user will not see breaking changes for sure.
Even this is not working sudo bin/logstash --path.settings=/etc/logstash -e 'input { stdin { } } output { stdout {} }'. It just sits there without showing me any msg like " Pipeline main started". If i type something like Hello, it just give me nothing and sit there

Logstash Error.png1011×69 1.19 KB

Vishal_Sharma1 · November 1, 2016, 4:45pm

so after running for 15 mins i got below
sudo bin/logstash --path.settings=/etc/logstash -e 'input { stdin { } } output { stdout {} }'
[sudo] password for vishne0:
Hell0 !
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties.
2016-11-01 12:39:45,782 Api Webserver ERROR No log4j2 configuration file found. Using default configuration: logging only errors to the console.
[vishne0@Elastic logstash]$ Hell0 !
-bash: Hell0: command not found

Vishal_Sharma1 · November 1, 2016, 4:47pm

In /var/log/logstash there is a file logstash-plain.log, it says below

[2016-11-01T12:39:45,739][ERROR][logstash.agent ] failed to fetch pipeline configuration {:message=>"No config files found: /etc/logstash/conf.d/*\nCan you make sure this path is a logstash config file?"}

now why it's looking at conf.d ? something is really really wrong in here.

theuntergeek · November 1, 2016, 5:14pm

We have an issue for that: https://github.com/elastic/logstash/issues/6062

We haven't figured out a way around it yet. It has to do with the fact that -e syntax was originally meant to be merged with the content of whatever config file you specified with -f so you could hypothetically have -e 'output { stdout { codec => rubydebug } } and just add stdout logging.

Now that configuration files and paths are specified in /etc/logstash/logstash.yml, and the packaged version specifies /etc/logstash/conf.d/*, that's why it's not working the way it did in older versions. We're still working on a resolution for this.

skeer · November 1, 2016, 5:22pm

Ran into the exact same issue. With regards to Logstash only so far, the current directions/tutorial/setup guide is not very conducive to noobs like myself.
https://www.elastic.co/guide/en/logstash/5.0/installing-logstash.html

Here I tried copying a few config files around but it didn't help

http://pastebin.com/smGtHszS

bhatch · November 1, 2016, 8:03pm

I was trying version 5.0 out yesterday and came across a similar issue.

5.0 forces you to have the settings file. But the default settings file has a row:
path.config: /etc/logstash/conf.d

This becomes a problem if you run the application and use -f to point it to a custom configuration file like this.
sudo /usr/share/logstash/bin/logstash -f /home/ec2-user/logstashfileinput.yaml --path.settings /etc/logstash/

Whatever file you point to using -f gets ignored and instead uses the file located at /etc/logstash/conf.d/. Assuming there even is one. The settings file takes precedence over whatever you pass in as a command line flag. If you were already planning on using that file then it isn't a problem, but for my cases I wasn't.
I haven't tested it, but it's possible that whatever is in the settings file would also take precedence over -w and -b too.

It probably won't affect anything on our production machines, but it's a bit annoying when I am trying to quickly test different configs.

skeer · November 1, 2016, 8:05pm

Let me ask a stupid question here since I am very green here. The config file that Logstash is wanting is really the yml for inputs/filters/outputs... right?

bhatch · November 1, 2016, 8:16pm

Correct.

There are now two "config" files.
The one that has been around forever is what holds the inputs, filters, and outputs. That is referenced by the -f command line flag.
The settings file introduced in 5.0 holds settings such as number of pipeline workers, batch size, config location, logging location, logging level, and other bits of info. The issue here is that these settings may exists as part of a command line flag, and also in the settings file.

skeer · November 1, 2016, 8:25pm

Now should teh default install have created a default config/yml file? Both locations on my system were empty.

theuntergeek · November 1, 2016, 8:36pm

We have a potential fix to allow -e to still be used: create an empty file in /etc/logstash/conf.d. Call it whatever you like, e.g.

sudo touch /etc/logstash/conf.d/empty.conf

Even though the file is empty, it should prevent this error:

Should this work for those trying to use -e to do the test config in the docs, we will put an empty config file in /etc/logstash/conf.d in future package releases.

@Vishal_Sharma1

It's right here in the breaking changes documentation.

theuntergeek · November 1, 2016, 8:37pm

Yes, it does create a default in /etc/logstash/logstash.yml. If that's empty, then something is wrong. The package provides a file filled with appropriate default values. Something or someone may have overwritten this file.

However, there is no default pipeline configuration in /etc/logstash/conf.d.

theuntergeek · November 1, 2016, 8:48pm

Scratch that. We may try a different approach. We're discussing it now.

skeer · November 1, 2016, 8:50pm

Ok gotcha, yes I have the default yml then. Sorry too much cmdline today
So I copied the logstash.yml and the log4j2.properties file into /usr/share/logstash/config just to make sure both possible config places were populated. Still the error persists.

theuntergeek · November 1, 2016, 8:55pm

@skeer, the --path.settings flag is still necessary if you're running from the command line. If it is omitted, it will fail. That's the error message you're seeing.

Try:

sudo -u logstash /usr/share/logstash/bin/logstash \
 --path.settings /etc/logstash \
 -f /path/to/first-pipeline.conf \
 --config.test_and_exit

I put it on multiple lines so you can see it easier. --path.settings is required, but in this case, -f overrides the --path.config in the settings file.