Logstash 5 not running


(Vishal Sharma) #1

hey there,

I am following the official document and seems like it's not working. I am using a freshly installed CentOS 7 system and have Java 1.8 installed. As per the document when i run below
[Foo@Elastic logstash]$ sudo bin/logstash -e 'input { stdin { } } output { stdout {} }'

It just stays like that no output. I have waited for 15 mins now. Any pointers why this is happening?
Thanks
V


Logstash package install -e not working
(Vishal Sharma) #2

Also after an long time i just typed Hello and below is what i have got

INFO: Logstash requires a setting file which is typically located in $LS_HOME/config or /etc/logstash. If you installed Logstash through a package and are starting it manually please specify the location to this settings file by passing in "--path.settings=/path/.."
ERROR: Failed to load settings file from "path.settings". Aborting... path.setting=/usr/share/logstash/config, exception=Errno::ENOENT, message=>No such file or directory

Why's that ?


(Amit Oren) #3

Well, as the document says - if you don't start logstash via the service, you should pass logstash.yml through the --path.settings i.e.
sudo bin/logstash --path.settings=/etc/logstash/logstash.yml -e 'input { stdin { } } output { stdout {} }'
Assuming you've installed it via the package.


(Vishal Sharma) #4

Ah really ? but it's not even mention in the official document that i need to include --path.settings=/etc/logstash/logstash.yml. Few things in here worth checking

  1. Why we need to provide --path.settings? it should be a default behavior of logstash to pick up the file from default location.
  2. Why it's not mentioned in the official document?
  3. Even running bin/logstash --path.settings=/etc/logstash/logstash.yml -e 'input { stdin { } } output { stdout {} }'
    is failing and giving me below error

INFO: Logstash requires a setting file which is typically located in $LS_HOME/config or /etc/logstash. If you installed Logstash through a package and are starting it manually please specify the location to this settings file by passing in "--path.settings=/path/.."
ERROR: Failed to load settings file from "path.settings". Aborting... path.setting=/etc/logstash/logstash.yml, exception=Errno::ENOENT, message=>No such file or directory - /etc/logstash/logstash.yml/logstash.yml

Regards
V


(Gabriel Tessier) #5

Hi,
I also have a similar problem following the official documentation, installing on an Amzon instance (CentOS 7).

The documentation is correct until "sudo yum install logstash".
After the command chkconfig --add logstash the documentation is not correct cause you need to be root to add it at boot so there's a sudo missing.
Even with sudo this command is not working the logstash bin is not found.

I expect that is the problem so that why Vishal_Sharma1 can't start it as service and try to run the bin.

Somebody can confirm that the yum installation is not working, can it be a rpm problem?
Cause I only copy paste the command and stop after the chkconfig --add logstash.

I checked in the /etc/init.d there's no logstash.

Thanks for any help.


(Amit Oren) #6

Huh, it seems like you actually need to give it the directory, not the file itself (No such file or directory - /etc/logstash/logstash.yml/logstash.yml)
Try changing it to --path.settings=/etc/logstash
It does seem the documentation is not 100% up to date - You can see the changes (and where they say you should specify --path.settings) at the breaking changes page


(Vishal Sharma) #7

ok let me clear few things in here

  1. I have installed the logstash from repo.
  2. There is no mention about path.settings in official doc.
  3. A new user will not see breaking changes for sure.
  4. Even this is not working sudo bin/logstash --path.settings=/etc/logstash -e 'input { stdin { } } output { stdout {} }'. It just sits there without showing me any msg like " Pipeline main started". If i type something like Hello, it just give me nothing and sit there

Elasticsearch output plugin does not write to elasticsearch
(Vishal Sharma) #8

so after running for 15 mins i got below
sudo bin/logstash --path.settings=/etc/logstash -e 'input { stdin { } } output { stdout {} }'
[sudo] password for vishne0:
Hell0 !
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties.
2016-11-01 12:39:45,782 Api Webserver ERROR No log4j2 configuration file found. Using default configuration: logging only errors to the console.
[vishne0@Elastic logstash]$ Hell0 !
-bash: Hell0: command not found


(Vishal Sharma) #9

In /var/log/logstash there is a file logstash-plain.log, it says below

[2016-11-01T12:39:45,739][ERROR][logstash.agent ] failed to fetch pipeline configuration {:message=>"No config files found: /etc/logstash/conf.d/*\nCan you make sure this path is a logstash config file?"}

now why it's looking at conf.d ? something is really really wrong in here.


(Aaron Mildenstein) #10

We have an issue for that: https://github.com/elastic/logstash/issues/6062

We haven't figured out a way around it yet. It has to do with the fact that -e syntax was originally meant to be merged with the content of whatever config file you specified with -f so you could hypothetically have -e 'output { stdout { codec => rubydebug } } and just add stdout logging.

Now that configuration files and paths are specified in /etc/logstash/logstash.yml, and the packaged version specifies /etc/logstash/conf.d/*, that's why it's not working the way it did in older versions. We're still working on a resolution for this.


#11

Ran into the exact same issue. With regards to Logstash only so far, the current directions/tutorial/setup guide is not very conducive to noobs like myself.
https://www.elastic.co/guide/en/logstash/5.0/installing-logstash.html

Here I tried copying a few config files around but it didn't help

http://pastebin.com/smGtHszS


(Brandon Hatch) #12

I was trying version 5.0 out yesterday and came across a similar issue.

5.0 forces you to have the settings file. But the default settings file has a row:
path.config: /etc/logstash/conf.d

This becomes a problem if you run the application and use -f to point it to a custom configuration file like this.
sudo /usr/share/logstash/bin/logstash -f /home/ec2-user/logstashfileinput.yaml --path.settings /etc/logstash/

Whatever file you point to using -f gets ignored and instead uses the file located at /etc/logstash/conf.d/. Assuming there even is one. The settings file takes precedence over whatever you pass in as a command line flag. If you were already planning on using that file then it isn't a problem, but for my cases I wasn't.
I haven't tested it, but it's possible that whatever is in the settings file would also take precedence over -w and -b too.

It probably won't affect anything on our production machines, but it's a bit annoying when I am trying to quickly test different configs.


#13

Let me ask a stupid question here since I am very green here. The config file that Logstash is wanting is really the yml for inputs/filters/outputs... right?


(Brandon Hatch) #14

Correct.

There are now two "config" files.
The one that has been around forever is what holds the inputs, filters, and outputs. That is referenced by the -f command line flag.
The settings file introduced in 5.0 holds settings such as number of pipeline workers, batch size, config location, logging location, logging level, and other bits of info. The issue here is that these settings may exists as part of a command line flag, and also in the settings file.


#15

Now should teh default install have created a default config/yml file? Both locations on my system were empty.


(Aaron Mildenstein) #16

We have a potential fix to allow -e to still be used: create an empty file in /etc/logstash/conf.d. Call it whatever you like, e.g.

sudo touch /etc/logstash/conf.d/empty.conf

Even though the file is empty, it should prevent this error:

Should this work for those trying to use -e to do the test config in the docs, we will put an empty config file in /etc/logstash/conf.d in future package releases.

@Vishal_Sharma1

It's right here in the breaking changes documentation.


(Aaron Mildenstein) #17

Yes, it does create a default in /etc/logstash/logstash.yml. If that's empty, then something is wrong. The package provides a file filled with appropriate default values. Something or someone may have overwritten this file.

However, there is no default pipeline configuration in /etc/logstash/conf.d.


(Aaron Mildenstein) #18

Scratch that. We may try a different approach. We're discussing it now.


#19

Ok gotcha, yes I have the default yml then. Sorry too much cmdline today :slight_smile:
So I copied the logstash.yml and the log4j2.properties file into /usr/share/logstash/config just to make sure both possible config places were populated. Still the error persists.


(Aaron Mildenstein) #20

@skeer, the --path.settings flag is still necessary if you're running from the command line. If it is omitted, it will fail. That's the error message you're seeing.

Try:

sudo -u logstash /usr/share/logstash/bin/logstash \
 --path.settings /etc/logstash \
 -f /path/to/first-pipeline.conf \
 --config.test_and_exit

I put it on multiple lines so you can see it easier. --path.settings is required, but in this case, -f overrides the --path.config in the settings file.