Logstash 5 not running

path permission issue? like what explain more

Well, it was simple. Logstash couldn't read the config file in /etc/logstash/conf.d because file itself was locked down. So, once I granted the file right permissions with chmod I was able to get the pipeline working etc.

Hope this helps

thats strange let me check.

It appears that my logstash is working when I use path.settings command but this is the error I get.

-bash-4.1$ sudo /usr/share/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }'
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs to console
The stdin plugin is now waiting for input:

What ended up working for you? I am having the same frustrations and I even get different errors in the log in successive attempts without even changing the command. Sometimes just randomly stopping "pipeline:id=>main" and other times with the no config conf.d error.

I did a 3Gig import last night and even using the same command today - without rebooting or changing anything, trying to do a different file, it won't work. And this file, rather than 3Gig is just about 200K. I have tried 'ln -s' and creating a file in conf.d, I've even 'ln -s' conf.d to /etc/logstash...

Can anyone shed some light on any updates? I am using the: " bin/logstash --path.settings=/etc/logstash -f [path/file] " command.

Hi Jon,

I think I ended up creating an empty config file as described here. I don't remember exactly because this setup was just for practice.

For the real implementation, we used Logstash and Elasticsearch via docker containers, here and here. Docker ended up being a very painless way to do this, and those containers are officially maintained by Elastic.

HTH!

Thanks for the reply and the Docker info!

you da man, Hans!

I got the same problem:confused:

Same problem. Did anyone find any workaround or solution?

Hi,
The issue is no more there with latest release. What version you are using?
Also make sure you run using it something like sudo bin/logstash -e
Regards
Vishal

Same trouble. Any benefit response? :sob:

The issue is due to few things

  1. Not enough permission to logstash user
  2. Problem with the grok of you conf file in /etc/logstash/conf.d
  3. Not enough data is being generated in the log file

Had problem with systemd running logstash. Manualy was ok (with sudo -u logstash /usr/share/logstash/bin/logstash --path.settings=/etc/logstash -f /etc/logstash/conf.d/ &), with startup script not. Tried different things on yum repository install on centos 7. So the only nice configuration for me was:

-clean install
-i put all my inputs to /etc/logstash/conf.d folder

  • i corrected logstash.service to ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash" "--path.config" "/etc/logstash/conf.d"
  • systemctl daemon-reload
  • systemctl start logstash

All other experiments with symbolic links and so on, were non working.

Hope this helps someone.

2 Likes

cool

This worked.
Location for logstash.service file on ubuntu 16.4 : /etc/systemd/system/logstash.service

I had the same issue. It was a permission problem when using custom grok patterns. Whichever user is running logstash needs read and execute permissions on the folder and files being used for this.

Hi everyone,

I really admire all the effort put here and I really hope the documentation was corrected, in my honest opinion it is wrong. I never got the -e '...' option to work with or without a /etc/logstash/conf.d/test.conf file.

Just to give other people an idea though what my workaround was, here are the commands that did not work and the one that finally worked for me (on Ubuntu LTS 16.04):

  1. Download Debian version of Logstash from here: https://www.elastic.co/downloads/logstash

  2. Install it: sudo dpkg -i logstash-6.1.1.deb

  3. No default permissions changed, I only have one sudo user in this machine (in my previous experience I remember Logstash required to be run as root or sudo user)

sudo ls -ltr /etc/logstash
total 24
-rw-r--r-- 1 root root 1659 Dec 17 16:48 startup.options
-rw-r--r-- 1 root root 1334 Dec 17 16:48 log4j2.properties
-rw-r--r-- 1 root root 1738 Dec 17 16:48 jvm.options
-rw-r--r-- 1 root root 6425 Jan 9 16:24 logstash.yml
drwxrwxr-x 2 root root 4096 Jan 9 17:02 conf.d

  1. Default contents of logstash.yml
    sudo grep -v ^# /etc/logstash/logstash.yml
    path.data: /var/lib/logstash
    path.config: /etc/logstash/conf.d/*.conf
    path.logs: /var/log/logstash

  2. Default permissions for /usr/share/logstash
    rwxrwxr-x 10 logstash logstash 4096 Jan 9 16:12 logstash
    sudo ls -ltr /usr/share/logstash/
    total 96
    -rw-r--r-- 1 logstash logstash 589 Dec 17 16:48 LICENSE
    -rw-r--r-- 1 logstash logstash 2276 Dec 17 16:48 CONTRIBUTORS
    -rw-r--r-- 1 logstash logstash 20956 Dec 17 16:50 Gemfile.lock
    -rw-r--r-- 1 logstash logstash 3809 Dec 17 16:51 Gemfile
    -rw-rw-r-- 1 logstash logstash 26953 Dec 17 16:51 NOTICE.TXT
    drwxrwxr-x 2 logstash logstash 4096 Dec 17 16:51 data
    drwxrwxr-x 3 logstash logstash 4096 Jan 9 16:12 tools
    drwxrwxr-x 4 logstash logstash 4096 Jan 9 16:12 vendor
    drwxrwxr-x 5 logstash logstash 4096 Jan 9 16:12 lib
    drwxrwxr-x 2 logstash logstash 4096 Jan 9 16:12 bin
    drwxrwxr-x 4 logstash logstash 4096 Jan 9 16:12 modules
    drwxrwxr-x 4 logstash logstash 4096 Jan 9 16:12 logstash-core
    drwxrwxr-x 3 logstash logstash 4096 Jan 9 16:12 logstash-core-plugin-api

  3. I only created the following file:
    cat /etc/logstash/conf.d/stdin-stdout.conf
    input {
    stdin { }
    }

output {
stdout {}
}

  1. I also manually created and set the permissions for this log file:
    ls -ltr /var/log
    drwxrwxr-x 2 logstash logstash 4096 Jan 9 16:45 logstash

ls -ltr /var/log/logstash/logstash-plain.log
-rw-rw-r-- 1 logstash logstash 14348 Jan 9 17:43 /var/log/logstash/logstash-plain.log

  1. Failed command:
    sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash -e 'input { stdin { } } output { stdout {} }'

Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
ERROR: Settings 'path.config' (-f) and 'config.string' (-e) can't be used simultaneously.

  1. Failed command:
    sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash

Sending Logstash's logs to ...
Exit with no output

  1. Failed command:
    cd /usr/share/logstash
    sudo -u logstash bin/logstash --path.settings /etc/logstash
    [sudo] password for ...:
    Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
    Exit with no output

  2. Failed command:
    cd /usr/share/logstash
    bin/logstash --path.settings /etc/logstash

Sending Logstash's logs to ...

2018-01-09 17:41:25,582 main ERROR FileManager (/var/log/logstash/logstash-plain.log) java.io.FileNotFoundException: /var/log/logstash/logstash-plain.log (Permission denied) java.io.FileNotFoundException: /var/log/logstash/logstash-plain.log (Permission denied)

  1. Failed command:
    cd /usr/share/logstash
    bin/logstash --path.settings /etc/logstash -e 'input { stdin { } } output { stdout {} }'

Sending Logstash's logs to ...

2018-01-09 18:11:28,473 main ERROR FileManager (/var/log/logstash/logstash-plain.log) java.io.FileNotFoundException: /var/log/logstash/logstash-plain.log (Permission denied)

  1. Failed command:
    cd /usr/share/logstash
    sudo bin/logstash --path.settings /etc/logstash -e 'input { stdin { } } output { stdout {} }'

Sending ...

ERROR: Settings 'path.config' (-f) and 'config.string' (-e) can't be used simultaneously.

  1. Failed command:
    cd ~
    /usr/share/logstash/bin/logstash --path.settings /etc/logstash

Sending ...

2018-01-09 18:14:57,517 main ERROR FileManager (/var/log/logstash/logstash-plain.log) java.io.FileNotFoundException: /var/log/logstash/logstash-plain.log (Permission denied)

  1. Finally a working command:
    cd ~
    sudo /usr/share/logstash/bin/logstash --path.settings /etc/logstash

Sending ...

The stdin plugin is now waiting for input:
I guess there is no -e option for logstash
2018-01-09T23:17:08.654Z Elasticsearch I guess there is no -e option for logstash
^C

  1. Finally a working command:
    cd /usr/share/logstash
    sudo bin/logstash --path.settings /etc/logstash

Sending ...

The stdin plugin is now waiting for input:
A good example of what works and what not with logstash is a must!
2018-01-09T22:42:58.931Z Elasticsearch A good example of what works and what not with logstash is a must!
^C

  1. Failed command:
    cd ~
    sudo mv /etc/logstash/conf.d/stdin-stdout.conf /tmp/

sudo /usr/share/logstash/bin/logstash --path.settings /etc/logstash

Sending ...

[ERROR] 2018-01-09 18:22:42.531 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] sourceloader - No configuration found in the configured sources.

  1. Failed command:
    cd ~
    sudo mv /etc/logstash/conf.d/stdin-stdout.conf /tmp/

sudo /usr/share/logstash/bin/logstash --path.settings /etc/logstash -e 'input { stdin { } } output { stdout {} }'
Sending ...
ERROR: Settings 'path.config' (-f) and 'config.string' (-e) can't be used simultaneously.

  1. Kinda Failed command:
    cd ~
    sudo mv /etc/logstash/conf.d/stdin-stdout.conf /tmp/

sudo /usr/share/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }'
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
The stdin plugin is now waiting for input:
what ?
2018-01-09T23:25:14.561Z Elasticsearch what ?

In the past in Logstash 5.6, I looked back at my notes and I see Logstash worked just fine as root but I can't remember if it worked with the -e option. Something tells me it did work:
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/some.conf --path.settings=/etc/logstash

My questions and many others I assume are:
Q1) What is the quickest way for a new user to use Logstash ?
Q2) What is the production recommended way to run Logstash and continually ingest data from an increasing number of data sources continually ?

Feel free to ask me any clarification question,
George

Look also at this topic:

and issues resolved:


The resoltion is not only do not use -f -e flag on command,but also you must have right conf in /etc/logstash/conf.d/. more important is the input config like 'input { stdin { } } ' can't start with systemctl .You may change a example config like blew:

input {
beats {
port => 5000
}
}
output { stdout {} }