Logstash 5 not running

[saikrishnagaddipati (krishna_gaddipati)]

Same problem. Did anyone find any workaround or solution?

[Vishal_Sharma1 (Vishal Sharma)]

Hi,
The issue is no more there with latest release. What version you are using?
Also make sure you run using it something like sudo bin/logstash -e
Regards
Vishal

[sehen1900 (Dua LİPA)]

Same trouble. Any benefit response?

[Vishal_Sharma1 (Vishal Sharma)]

The issue is due to few things

1. Not enough permission to logstash user
2. Problem with the grok of you conf file in /etc/logstash/conf.d
3. Not enough data is being generated in the log file

[pingwin (Tinko)]

Had problem with systemd running logstash. Manualy was ok (with sudo -u logstash /usr/share/logstash/bin/logstash --path.settings=/etc/logstash -f /etc/logstash/conf.d/ &), with startup script not. Tried different things on yum repository install on centos 7. So the only nice configuration for me was:

-clean install
-i put all my inputs to /etc/logstash/conf.d folder

• i corrected logstash.service to ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash" "--path.config" "/etc/logstash/conf.d"
• systemctl daemon-reload
• systemctl start logstash

All other experiments with symbolic links and so on, were non working.

Hope this helps someone.

[Vishal_Sharma1 (Vishal Sharma)]

cool

[softenter (Prashant Kumar)]

This worked.
Location for logstash.service file on ubuntu 16.4 : /etc/systemd/system/logstash.service

[ffernandez (Fernando)]

I had the same issue. It was a permission problem when using custom grok patterns. Whichever user is running logstash needs read and execute permissions on the folder and files being used for this.

[georgezoto]

Hi everyone,

I really admire all the effort put here and I really hope the documentation was corrected, in my honest opinion it is wrong. I never got the -e '...' option to work with or without a /etc/logstash/conf.d/test.conf file.

Just to give other people an idea though what my workaround was, here are the commands that did not work and the one that finally worked for me (on Ubuntu LTS 16.04):

1. Download Debian version of Logstash from here: https://www.elastic.co/downloads/logstash

2. Install it: sudo dpkg -i logstash-6.1.1.deb

3. No default permissions changed, I only have one sudo user in this machine (in my previous experience I remember Logstash required to be run as root or sudo user)

sudo ls -ltr /etc/logstash
total 24
-rw-r--r-- 1 root root 1659 Dec 17 16:48 startup.options
-rw-r--r-- 1 root root 1334 Dec 17 16:48 log4j2.properties
-rw-r--r-- 1 root root 1738 Dec 17 16:48 jvm.options
-rw-r--r-- 1 root root 6425 Jan 9 16:24 logstash.yml
drwxrwxr-x 2 root root 4096 Jan 9 17:02 conf.d

4. Default contents of logstash.yml
   sudo grep -v ^# /etc/logstash/logstash.yml
   path.data: /var/lib/logstash
   path.config: /etc/logstash/conf.d/*.conf
   path.logs: /var/log/logstash

5. Default permissions for /usr/share/logstash
   rwxrwxr-x 10 logstash logstash 4096 Jan 9 16:12 logstash
   sudo ls -ltr /usr/share/logstash/
   total 96
   -rw-r--r-- 1 logstash logstash 589 Dec 17 16:48 LICENSE
   -rw-r--r-- 1 logstash logstash 2276 Dec 17 16:48 CONTRIBUTORS
   -rw-r--r-- 1 logstash logstash 20956 Dec 17 16:50 Gemfile.lock
   -rw-r--r-- 1 logstash logstash 3809 Dec 17 16:51 Gemfile
   -rw-rw-r-- 1 logstash logstash 26953 Dec 17 16:51 NOTICE.TXT
   drwxrwxr-x 2 logstash logstash 4096 Dec 17 16:51 data
   drwxrwxr-x 3 logstash logstash 4096 Jan 9 16:12 tools
   drwxrwxr-x 4 logstash logstash 4096 Jan 9 16:12 vendor
   drwxrwxr-x 5 logstash logstash 4096 Jan 9 16:12 lib
   drwxrwxr-x 2 logstash logstash 4096 Jan 9 16:12 bin
   drwxrwxr-x 4 logstash logstash 4096 Jan 9 16:12 modules
   drwxrwxr-x 4 logstash logstash 4096 Jan 9 16:12 logstash-core
   drwxrwxr-x 3 logstash logstash 4096 Jan 9 16:12 logstash-core-plugin-api

6. I only created the following file:
   cat /etc/logstash/conf.d/stdin-stdout.conf
   input {
   stdin { }
   }

output {
stdout {}
}

7. I also manually created and set the permissions for this log file:
   ls -ltr /var/log
   drwxrwxr-x 2 logstash logstash 4096 Jan 9 16:45 logstash

ls -ltr /var/log/logstash/logstash-plain.log
-rw-rw-r-- 1 logstash logstash 14348 Jan 9 17:43 /var/log/logstash/logstash-plain.log

8. Failed command:
   sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash -e 'input { stdin { } } output { stdout {} }'

Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
ERROR: Settings 'path.config' (-f) and 'config.string' (-e) can't be used simultaneously.

9. Failed command:
   sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash

Sending Logstash's logs to ...
Exit with no output

10. Failed command:
    cd /usr/share/logstash
    sudo -u logstash bin/logstash --path.settings /etc/logstash
    [sudo] password for ...:
    Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
    Exit with no output

11. Failed command:
    cd /usr/share/logstash
    bin/logstash --path.settings /etc/logstash

Sending Logstash's logs to ...

2018-01-09 17:41:25,582 main ERROR FileManager (/var/log/logstash/logstash-plain.log) java.io.FileNotFoundException: /var/log/logstash/logstash-plain.log (Permission denied) java.io.FileNotFoundException: /var/log/logstash/logstash-plain.log (Permission denied)

12. Failed command:
    cd /usr/share/logstash
    bin/logstash --path.settings /etc/logstash -e 'input { stdin { } } output { stdout {} }'

Sending Logstash's logs to ...

2018-01-09 18:11:28,473 main ERROR FileManager (/var/log/logstash/logstash-plain.log) java.io.FileNotFoundException: /var/log/logstash/logstash-plain.log (Permission denied)

13. Failed command:
    cd /usr/share/logstash
    sudo bin/logstash --path.settings /etc/logstash -e 'input { stdin { } } output { stdout {} }'

Sending ...

ERROR: Settings 'path.config' (-f) and 'config.string' (-e) can't be used simultaneously.

14. Failed command:
    cd ~
    /usr/share/logstash/bin/logstash --path.settings /etc/logstash

Sending ...

2018-01-09 18:14:57,517 main ERROR FileManager (/var/log/logstash/logstash-plain.log) java.io.FileNotFoundException: /var/log/logstash/logstash-plain.log (Permission denied)

15. Finally a working command:
    cd ~
    sudo /usr/share/logstash/bin/logstash --path.settings /etc/logstash

Sending ...

The stdin plugin is now waiting for input:
I guess there is no -e option for logstash
2018-01-09T23:17:08.654Z Elasticsearch I guess there is no -e option for logstash
^C

16. Finally a working command:
    cd /usr/share/logstash
    sudo bin/logstash --path.settings /etc/logstash

Sending ...

The stdin plugin is now waiting for input:
A good example of what works and what not with logstash is a must!
2018-01-09T22:42:58.931Z Elasticsearch A good example of what works and what not with logstash is a must!
^C

17. Failed command:
    cd ~
    sudo mv /etc/logstash/conf.d/stdin-stdout.conf /tmp/

sudo /usr/share/logstash/bin/logstash --path.settings /etc/logstash

Sending ...

[ERROR] 2018-01-09 18:22:42.531 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] sourceloader - No configuration found in the configured sources.

18. Failed command:
    cd ~
    sudo mv /etc/logstash/conf.d/stdin-stdout.conf /tmp/

sudo /usr/share/logstash/bin/logstash --path.settings /etc/logstash -e 'input { stdin { } } output { stdout {} }'
Sending ...
ERROR: Settings 'path.config' (-f) and 'config.string' (-e) can't be used simultaneously.

19. Kinda Failed command:
    cd ~
    sudo mv /etc/logstash/conf.d/stdin-stdout.conf /tmp/

sudo /usr/share/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }'
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
The stdin plugin is now waiting for input:
what ?
2018-01-09T23:25:14.561Z Elasticsearch what ?

In the past in Logstash 5.6, I looked back at my notes and I see Logstash worked just fine as root but I can't remember if it worked with the -e option. Something tells me it did work:
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/some.conf --path.settings=/etc/logstash

My questions and many others I assume are:
Q1) What is the quickest way for a new user to use Logstash ?
Q2) What is the production recommended way to run Logstash and continually ingest data from an increasing number of data sources continually ?

Feel free to ask me any clarification question,
George

[georgezoto]

Look also at this topic:

Logstash requires setting a file

Logstash Info error: A setting file is missing I installed the deb file of logstash 5.0 and I starting logstash for the first time. I get an INFO error: INFO: Logstash requires a setting file which is typically located in $LS_HOME/config or /etc/logstash. If you installed Logstash through a package and are starting it manually please specify the location to this settings file by passing in "--path.settings=/path/.." ERROR: Failed to load settings file from "path.settings". Aborting... path.s…

and issues resolved:

Logstash installed via rpm and apt packages cannot be manually started with `-e` flag

-e and -f should merge even without valid conf file

[strengthening]

@georgezoto

input { stdin { } } output { stdout {} }

The resoltion is not only do not use -f -e flag on command,but also you must have right conf in /etc/logstash/conf.d/. more important is the input config like 'input { stdin { } } ' can't start with systemctl .You may change a example config like blew:

input {
beats {
port => 5000
}
}
output { stdout {} }

[dessy53 (desmond)]

my services are up; logstash started yet when i run the below command;

i am still getting no such directory can not read, pls what could be the way out,

my /etc/logstash/elastiflow/conf.d/*.conf"