Logstash 5 alpha5-1 - Failed to load settings file from "path.settings". Aborting

rwagner · August 18, 2016, 2:25pm

I'm getting the following error when try run:

# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf
--- jar coordinate com.fasterxml.jackson.core:jackson-annotations already loaded with version 2.7.1 - omit version 2.7.0
--- jar coordinate com.fasterxml.jackson.core:jackson-databind already loaded with version 2.7.1 - omit version 2.7.1-1
Logstash has a new settings file which defines start up time settings. This file is typically located in $LS_HOME/config or /etc/logstash. If you installed Logstash through a package and are starting it manually please specify the location to this settings file by passing in "--path.settings=/path/.." in the command line options {:level=>:warn}

Failed to load settings file from "path.settings". Aborting... {"path.settings"=>"/usr/share/logstash/config", "exception"=>Errno::ENOENT, "message"=>"No such file or directory - /usr/share/logstash/config/logstash.yml", :level=>:fatal}

JAVA
#java -version
openjdk version "1.8.0_101"

Logstash version

#yum list installed | grep logstash
logstash.noarch                        1:5.0.0~alpha5-1                @logstash-5.0

Plugin beats

# /usr/share/logstash/bin/logstash-plugin list | grep beat
logstash-input-beats

Logstash conf

# cat /etc/logstash/conf.d/logstash.conf
input {
  beats  {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

I'm using CentOS 7.

#cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)

Someone can help?

theuntergeek · August 18, 2016, 2:53pm

The error message you pasted explains what went wrong and how to fix it. I'll re-paste what you have above and add line breaks to make it easier to read without scrolling left to right (I'm not trying to be patronizing. I'm trying to make it easier for future readers to see the issue and understand the fix):

Logstash has a new settings file which defines start up time settings. 

This file is typically located in $LS_HOME/config or /etc/logstash. 

If you installed Logstash through a package and are starting it manually 

please specify the location to this settings file by passing in 

"--path.settings=/path/.." in the command line options {:level=>:warn}

Failed to load settings file from "path.settings". 

Aborting... {"path.settings"=>"/usr/share/logstash/config", 
"exception"=>Errno::ENOENT, 
"message"=>"No such file or directory - /usr/share/logstash/config/logstash.yml", 
:level=>:fatal}

What this means is that you should start Logstash with a manually configured --path.settings, like this:

Before:

# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf

After:

# /usr/share/logstash/bin/logstash --path.settings /etc/logstash -f /etc/logstash/conf.d/logstash.conf

rwagner · August 18, 2016, 5:05pm

Ok, Really work! Thanks!

But, can i do for the logstash work normally, when i run with "service logstash start"?

My instalation was done with yum, using the following repo:

[logstash-5.0]
name=Logstash repository for 5.0.x packages
baseurl=https://packages.elastic.co/logstash/5.0/centos
gpgcheck=1
gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1

I haved put the path.settings configuration in "/etc/logstash/logstash.yml" and tried run "service logstash start" but dont work.

My /etc/logstash/logstash.yml:

path.data: /usr/share/logstash
path.config: /etc/logstash/conf.d
path.log: /var/log/logstash/logstash.log
path.settings: /etc/logstash

Other thing, the port is up under IPV6.

'# netstat -ntl | grep 5044
tcp6 0 0 :::5044 :::* list

Do you know where i can configure for IPV4?

theuntergeek · August 18, 2016, 5:21pm

What version of RHEL/CentOS are you using? That may determine how Logstash needs to be started.

Did you edit /etc/logstash/logstash.yml? You shouldn't have needed to change the path.settings attribute as it was preconfigured to point there already.

input {
  beats  {
    host => "127.0.0.1"
    port => 5044
  }
}

That should do IPv4.

rwagner · August 18, 2016, 5:47pm

Yes, i haved put the path.settings attribute after you say to run with this attribute, i try put in the logstash.yml configurations file, thinking it would work, but no.

theuntergeek · August 18, 2016, 7:20pm

That's why it isn't working. You're using upstart syntax and CentOS 7 is systemctl. Logstash 5 is auto-detecting the system init type and creating startup files accordingly. It just so happens that systemctl supersedes upstart, so it's the default on RHEL/CentOS 7.

See https://www.elastic.co/guide/en/logstash/5.0/running-logstash.html:

sudo systemctl start logstash.service

rwagner · August 19, 2016, 1:24pm

Hi Aaron,

In this way logstash really runs, but apparently he ignores the configuration file in the /etc/logstash/conf.d/logstash.conf , because the configured port for the beats don't up.

If i run in this way /usr/share/logstash/bin/logstash --path.settings /etc/logstash -f /etc/logstash/conf.d/logstash.conf, the port up.

The configuration is:

theuntergeek · August 19, 2016, 2:40pm

Without some -v verbose logs (or possibly debug logs), I won't be able to know what's going on. You edited the default files, so I can't say for sure whether that caused it or not. It should have picked up anything ending in .conf in /etc/logstash/conf.d. This is being used by many people everywhere right now, so I'm keen to know why it's not working for you.

rwagner · August 19, 2016, 4:42pm

I tried install all again. I haved install the CentOS vm again and the logstash from aforementioned yum repo, and the problem continues.

Below, the logstash running in two ways that I said:

Executed by systemctl

Executed by command:

I only have one .conf file in the /etc/logstash/conf.d.

The -v option don't work in logstash. If you want reproduce, you can install CentOS (install java by yum) and install the logstash alpha5 by the yum repo.

theuntergeek · August 19, 2016, 5:38pm

I don't know what's going on in your box, but a freshly spun up vagrant with CentOS 7 yields this:

[vagrant@localhost logstash]$ cat /etc/logstash/logstash.yml  | grep -v ^#
path.data: /var/lib/logstash
path.config: /etc/logstash/conf.d
path.log: /var/log/logstash/logstash.log

[vagrant@localhost logstash]$ cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)

[vagrant@localhost logstash]$ yum list installed | grep logstash
logstash.noarch                   1:5.0.0~alpha5-1               @logstash-5.0

[vagrant@localhost logstash]$ cat /etc/logstash/conf.d/logstash.conf
input {
  beats  {
    port => 5044
  }
}

output {
  stdout { codec => rubydebug }
}

[vagrant@localhost logstash]$ netstat -ntl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN
tcp6       0      0 :::5044                 :::*                    LISTEN
tcp6       0      0 :::22                   :::*                    LISTEN
tcp6       0      0 ::1:25                  :::*                    LISTEN
tcp6       0      0 127.0.0.1:9600          :::*                    LISTEN

[vagrant@localhost logstash]$ systemctl status logstash
● logstash.service - logstash
   Loaded: loaded (/etc/systemd/system/logstash.service; disabled; vendor preset: disabled)
   Active: active (running) since Fri 2016-08-19 17:31:57 UTC; 4min 1s ago
 Main PID: 30241 (java)
   CGroup: /system.slice/logstash.service
           └─30241 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -Xm...

Aug 19 17:31:57 localhost.localdomain systemd[1]: Started logstash.
Aug 19 17:31:57 localhost.localdomain systemd[1]: Starting logstash...
Aug 19 17:32:23 localhost.localdomain logstash[30241]: --- jar coordinate com.fasterxml.jackson.core:jackson-annotations already loaded with version 2.7.1 - omit version 2.7.0
Aug 19 17:32:23 localhost.localdomain logstash[30241]: --- jar coordinate com.fasterxml.jackson.core:jackson-databind already loaded with version 2.7.1 - omit version 2.7.1-1
Aug 19 17:32:28 localhost.localdomain logstash[30241]: --- jar coordinate com.fasterxml.jackson.core:jackson-core already loaded with version 2.7.1 - omit version 2.7.5
Aug 19 17:32:28 localhost.localdomain logstash[30241]: --- jar coordinate com.fasterxml.jackson.core:jackson-annotations already loaded with version 2.7.1 - omit version 2.7.5
Aug 19 17:32:28 localhost.localdomain logstash[30241]: --- jar coordinate com.fasterxml.jackson.core:jackson-databind already loaded with version 2.7.1 - omit version 2.7.5
Aug 19 17:32:28 localhost.localdomain logstash[30241]: --- jar coordinate com.fasterxml.jackson.module:jackson-module-afterburner already loaded with version 2.7.1 - omit version 2.7.5
Aug 19 17:32:29 localhost.localdomain logstash[30241]: ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
Aug 19 17:32:31 localhost.localdomain logstash[30241]: {:timestamp=>"2016-08-19T17:32:31.468000+0000", :message=>"Pipeline main started"}

As you can see, the only thing I've changed is the Logstash output (no elasticsearch, just stdout). I get the correct port running for Beats.

Again, I don't know what's going on inside your box, but I can't reproduce your problem locally

rwagner · August 19, 2016, 6:29pm

Hey Aaron, i think which the problem is something with permission. I saw that you were running with a non root user, and i intall all whith root. Then i restored a snapshot from the VM and add a new user(non root) and install all using sudo(e.g sudo yum intall logstash), thus it functioned.

If you can try install all with a root user, i think which you can reproduce.

But, for now, it's work.

theuntergeek · August 19, 2016, 6:45pm

The commands needed with root or sudo were anything yum, e.g. sudo yum install logstash, and systemctl, e.g. sudo systemctl start logstash. And of course, to edit the config file. I just used sudo, nothing special.

rwagner · August 19, 2016, 7:05pm

So, i understood what you said! What I said was that, when i installed, there was only the root user on the host. In this way it was installed, runned and edited all with the root user.

I believe that in this way you can reproduce, instaling and running all with root user, without using sudo(e.g # yum install logstash or # systemctl start logstash).

theuntergeek · August 19, 2016, 7:39pm

That shouldn't matter. To leave no stone unturned, I tried what you described. It still worked for me, however, it took longer to finish launching:

[root@localhost logstash]# systemctl status logstash
● logstash.service - logstash
   Loaded: loaded (/etc/systemd/system/logstash.service; disabled; vendor preset: disabled)
   Active: active (running) since Fri 2016-08-19 19:21:14 UTC; 4min 10s ago
 Main PID: 30861 (java)
   CGroup: /system.slice/logstash.service
           └─30861 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -Xm...

Aug 19 19:21:14 localhost.localdomain systemd[1]: Started logstash.
Aug 19 19:21:14 localhost.localdomain systemd[1]: Starting logstash...
Aug 19 19:21:39 localhost.localdomain logstash[30861]: --- jar coordinate com.fasterxml.jackson.core:jackson-annotations already loaded with version 2.7.1 - omit version 2.7.0
Aug 19 19:21:39 localhost.localdomain logstash[30861]: --- jar coordinate com.fasterxml.jackson.core:jackson-databind already loaded with version 2.7.1 - omit version 2.7.1-1
Aug 19 19:22:19 localhost.localdomain logstash[30861]: --- jar coordinate com.fasterxml.jackson.core:jackson-core already loaded with version 2.7.1 - omit version 2.7.5
Aug 19 19:22:19 localhost.localdomain logstash[30861]: --- jar coordinate com.fasterxml.jackson.core:jackson-annotations already loaded with version 2.7.1 - omit version 2.7.5
Aug 19 19:22:19 localhost.localdomain logstash[30861]: --- jar coordinate com.fasterxml.jackson.core:jackson-databind already loaded with version 2.7.1 - omit version 2.7.5
Aug 19 19:22:19 localhost.localdomain logstash[30861]: --- jar coordinate com.fasterxml.jackson.module:jackson-module-afterburner already loaded with version 2.7.1 - omit version 2.7.5
Aug 19 19:22:20 localhost.localdomain logstash[30861]: ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
Aug 19 19:22:21 localhost.localdomain logstash[30861]: {:timestamp=>"2016-08-19T19:22:21.254000+0000", :message=>"Pipeline main started"}

[root@localhost logstash]# netstat -ntl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN
tcp6       0      0 :::5044                 :::*                    LISTEN
tcp6       0      0 :::22                   :::*                    LISTEN
tcp6       0      0 ::1:25                  :::*                    LISTEN
tcp6       0      0 127.0.0.1:9600          :::*                    LISTEN

When launched via sudo (see my earlier post), it took 26 seconds after launch before the jar messages start, and then another 8 seconds before Pipeline main started. When launched as root, it took 25 seconds after launch before the jar messages start, and then an additional 42 seconds before Pipeline main started.

I do not think this has anything to do with Logstash, as Logstash has already been started as the Logstash user, and both implementations seem to have reached the jar loading section at the same time, so permissions have already been set. I could be wrong, but I do not know what could be done by Logstash differently. It's not like there's a flag somewhere that detected the user which started the systemctl process that spawned the Logstash process.

rwagner · August 21, 2016, 1:05pm

Well Aaron, I do not know what happened, but happened.
Sorry about not reproduce the situation.

Anyway, sorry for whatever and thanks for the help!

And the following configuration don't work for the IPV6 problem: [quote="theuntergeek, post:4, topic:58360"]
host => "127.0.0.1"
[/quote]

What worked for me was, i have put the following configuration in the /etc/sysctl.conf file:

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

Again, thank you for being so solicitous!

andySFt · March 19, 2017, 7:50pm

Have you find out what the problem was?