Thanks! So I used upstart (sudo initctl start logstash) since I have ubuntu 14.04. The ps aux | grep logstash command gave the following output: logstash 26825 121 9.2 3441576 375020 ? SNsl 14:54 0:16 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -Xmx1g -Xms256m -Xss2048k -Djffi.boot.library.path=/usr/share/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/usr/share/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/share/logstash/vendor/jruby -Djruby.lib=/usr/share/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /usr/share/logstash/lib/bootstrap/environment.rb logstash/runner.rb --path.settings /etc/logstash.
It doesn't seem like a performance problem for me since CPU usage is close to 0% and memory usage is comparable to other machines (most of the memory is used by elasticsearch and logstash). The strange thing is that once logstash starts parsing goes quickly but it takes too long to start (around one hour from entering the start command).
It seems like it is up and running... And I had the same problem the other day once I started logstash, it took around 5-10 minutes to get logstash to run and was using 48% of my cpu and I had only 8 Gigs of memory left. Switched to 12 GRAM and 100 GB of disk space with 2 CPUs and it went down to 20 seconds to initializing my config file. What is the CPU usage during your initialization of your config file?
Actually when it's running then CPU usage is more than 100% (which means I assume both processors are running).
While this can be from insufficient memory, you may actually be suffering from a lack of entropy. We're improving the documentation about this, in this ticket. See the JRuby Wiki entry about the same issue. People who have followed these instructions have cut 10 minute start times to less than 20 seconds.
Not startup but systemd in Centos 7, you should type the command
sudo systemctl enable logstash.service
if you want Logstash to be started when your computer startup.
Thanks for the valuable info! This looks to have solved my problem as well.
ok so my original question is still unanswered. What is the method to run logstash from command line as mentioned in the official document at https://www.elastic.co/guide/en/logstash/current/first-event.html,
if i have installed it from repo?. Also there are no changes in the official document about the issue. People keep talking about breaking change, but think about someone who just landed to the official document and trying to run logstash for the first time.
HI, I also use Centos 7 for all my installations and follow the following procedure for installing logstash and running logstash:
Install
https://www.elastic.co/guide/en/logstash/current/installing-logstash.html#package-repositories
run the following commands:
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
vi /etc/yum.repos.d/logstash.repo
paste the Repo information
sudo yum install logstash -y
After I add the configuration file to the following directory:
/etc/logstash/conf.d/file.conf
Now it is just to run the configuration file under test with the following command:
sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash -f /etc/logstash/conf.d/file.conf --config.test_and_exit
Usually I also have a second window open and follow the logs to see if there are any errors:
tail -f /var/log/logstash/logstash-plain.log
After this you just remove the last part "--config.test_and_exit" and add "&" to continue running this. The process will not start automatically when stopped or restarted, if this is required you can look at some sort of crontab config.
Hope this helps
well my exact question was in official documents just after the installation from the repo it says run below
bin/logstash -e 'input { stdin { } } output { stdout {} }' and this is not working. Did you get what i mean?
Hi theuntergeek, I understand that it's not simple to fix this issue quickly, but have you considered updating the documentation? I ran into this issue while following the instructions at Stashing Your First Event after installing from a package manager, and I've spent way too long trying to get this working.
I don't want to come off as a jerk here, but honestly, if I weren't doing this for work, I'd have already given up on logstash and started looking for another system.
I had the same exact issue except that it would just hang and would not produce any output. I spent enormous amount of time going through the documentation trying to figure out what the problem was. In the logs it was saying that it can't find the conf file in /etc/logstash/conf.d even though there was one. It ended up being the path permissions problem. As soon as I changed it it worked.
path permission issue? like what explain more
Well, it was simple. Logstash couldn't read the config file in /etc/logstash/conf.d because file itself was locked down. So, once I granted the file right permissions with chmod I was able to get the pipeline working etc.
Hope this helps
thats strange let me check.
It appears that my logstash is working when I use path.settings command but this is the error I get.
-bash-4.1$ sudo /usr/share/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }'
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs to console
The stdin plugin is now waiting for input:
What ended up working for you? I am having the same frustrations and I even get different errors in the log in successive attempts without even changing the command. Sometimes just randomly stopping "pipeline:id=>main" and other times with the no config conf.d error.
I did a 3Gig import last night and even using the same command today - without rebooting or changing anything, trying to do a different file, it won't work. And this file, rather than 3Gig is just about 200K. I have tried 'ln -s' and creating a file in conf.d, I've even 'ln -s' conf.d to /etc/logstash...
Can anyone shed some light on any updates? I am using the: " bin/logstash --path.settings=/etc/logstash -f [path/file] " command.
Hi Jon,
I think I ended up creating an empty config file as described here. I don't remember exactly because this setup was just for practice.
For the real implementation, we used Logstash and Elasticsearch via docker containers, here and here. Docker ended up being a very painless way to do this, and those containers are officially maintained by Elastic.
HTH!
Thanks for the reply and the Docker info!
you da man, Hans!
I got the same problem:confused: