Hello
I'm trying to install Logstash 5 on AWS Amazon Linux. The install is successful, however, the service does not exist. (I used both the rpm install and the repo with yum).
It' s the latest AMI (2016.09, running RPM 4.x
Can you help?
Thx
1 Like
Same issue - installed 5.0 via
sudo yum install logstash
no problems - but no service was created.
When executing
/usr/share/logstash/bin/logstash -help
.. nothing
/usr/share/logstash/bin/logstash -V
returned logstash 5.0.0
So something with the ruby/jruby setup... is my guess
ruby -v
ruby 2.0.0p648 (2015-12-16) [x86_64-linux]
java -version
java version "1.8.0_91"
Java(TM) SE Runtime Environment (build 1.8.0_91-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)
Any suggestions / hints ??
... a few missing points in the doc:
you need to execute
sudo /usr/share/logstash/bin/system-install
which uses the settings in
/etc/logstash/startup.options
in that file you need to change
LS_SETTINGS_DIR="/etc/logstash"
... but even though it says:
Successfully created system startup script for Logstash
there is no /etc/init.d/logstash
and /usr/share/logstash/bin/logstash -help keeps hanging
1 Like
Has anyone found a solution to this? I too am having trouble getting a proper init script to work for the new Logstash 5
Logstash detects which startup system is used and deployed the appropriate script. Please see https://www.elastic.co/guide/en/logstash/current/running-logstash.html
Yes I am aware how to manually start the application and run it in the background. What we are trying to say is that the provided installation documentation / install does not provide a way for system administrators to use the standard unix commands to verify from a monitoring solution/command line that a service is running. aka service logstash status .
Something we have had in previous packaged rpm installs.
It's still based on what I sent:
For upstart (CentOS/RHEL 6 based systems):
sudo initctl status logstash
For systemd, (CentOS/RHEL 7 based systems):
sudo systemctl status logstash
service is centered around SysV init-style startups. That's why it's not working for you. Logstash must not have installed a SysV init-style startup script, but rather one of the other kinds. Whatever flavor Amazon Linux is, that will be your command-style.
1 Like
Hi,
Same problem here, after running yum install logstash -y
you need to go to:
/usr/share/logstash/bin
you can test your installation like explain here:
running the following command:
./logstash -e 'input { stdin { } } output { stdout {} }'
https://www.elastic.co/guide/en/logstash/current/first-event.html
You'll have a big stacktrace so you need to:
1- there's no config file, you need to create a config dir and an empty logstash.yml file (read on other post empty file is ok) [don't forget to give the correct permission sudo chown logstatsh:logstash logstash.yml]
2- Then you need to create a log4j2.properties file, here also empty looks fine didn't find anything about on the forum.
3- Last one is the data directory I put 777 to make it work <--- this one I'm not sure.
After that you can run again the command and it will work!!!
./logstash -e 'input { stdin { } } output { stdout {} }'
Somebody fix it the same way? is there some already made log4j... file and what is the correct chmod for data?
Hope it can help.
Yes, I can confirm sudo initctl status logstash work for me!!!
Big thanks to theuntergeek !!
If you installed via package, you do not need to do this. It was done for you at installation.
This is because your system also has upstart or systemd, and Logstash will use the newer startup system it finds.
I'm not sure what may be going on. Being that this thread is about Amazon Linux (and therefore a VM/cloud system with shared resources), it could be a lack of entropy. When the sources of entropy are insufficient, JRuby hangs until the entropy pool is "full enough" to proceed. See: Improving startup time · jruby/jruby Wiki · GitHub, and the article it links, How to Setup Additional Entropy for Cloud Servers Using Haveged | DigitalOcean. Users who have employed this method have fixed 10 minute start times to be less than 20 seconds.
For me, in a non-entropy-boosted CentOS 6 Vagrant instance:
time /usr/share/logstash/bin/logstash --help
Usage:
bin/logstash [OPTIONS]
...
real 1m3.317s
user 0m30.644s
sys 0m1.254s
And on identical hardware, but not virtualized, using Ubuntu 16.04:
real 0m8.230s
user 0m29.252s
sys 0m0.956s
1 minute 3 seconds on a throttled VM, vs. 8.2 seconds on bare metal.
1 Like
@gabriel_tessier
You shouldn't need to go to such lengths to test your configuration.
sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash -t
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties.
Configuration OK
This will effectively test the configuration files in /etc/logstash/conf.d
That worked for me on AWS Amazon Linux. Just for starting it. I cannot use the "services" instruction in CloudFormation (this is what I was trying to achieve). I'm putting that in a "commands" instruction instead.
Thx anyway for helping with that (temporary) patch, still waiting for it to be a real service (as was previous version)
This was not created to be a temporary patch, as upstart and systemd are newer and much more robust. If you want to continue to use SysV init-style startup on a system that supports upstart or systemd, you will be having to manually override it. We plan on adding functionality for users to force Logstash to install SysV init style scripts, if desired. Otherwise, Logstash will continue to detect and install the more up-to-date startup methods.
I am using AWS, and more preciselly Amazon Linux 2016-09 machine.
It' s not a regular CentOs.
Also, I'm using CloudFormation (from AWS) also to create my servers and install Logstash. So, except by patching more the machine, I don't have a choice on what system to use to start the service.
I you plan to add it, that will be great.
Thx
All I am pointing out is that based on the documentation I see, and the output of the installation process, the expected behavior of Logstash is different than any of the other ELK stack services.
Amazon Linux AMI release 2016.09
sudo yum list installed | grep elastic
elasticsearch.noarch 5.0.0-1 @elasticsearch-5.x
filebeat.x86_64 5.0.0-1 @elastic-5.x
kibana.x86_64 5.0.0-1 @elastic-5.x
logstash.noarch 1:5.0.0-1 @elastic-5.x
metricbeat.x86_64 5.0.0-1 @elastic-5.x
packetbeat.x86_64 5.0.0-1 @elastic-5.x
Not sure if the 1:5.0.0-1 is significant vs. all the other components, but it is all from repo installs...
With these all at the same version I receive the following results on same system:
Elasticsearch
$ sudo service elasticsearch status
elasticsearch (pid 18607) is running...
$ sudo initctl status elasticsearch
initctl: Unknown job: elasticsearch
Kibana
$sudo service kibana status
kibana is running
$ sudo initctl status kibana
initctl: Unknown job: kibana
Packetbeat
$ sudo service packetbeat status
packetbeat-god (pid 3699) is running...
$ sudo initctl status packetbeat
initctl: Unknown job: packetbeat
Metricbeat
$ sudo service metricbeat status
metricbeat-god (pid 24120) is running...
$ sudo initctl status metricbeat
initctl: Unknown job: metricbeat
Filebeat
$ service filebeat status
filebeat-god (pid 3571) is running...
$ sudo initctl status filebeat
initctl: Unknown job: filebeat
Logstash
$ service logstash status
logstash: unrecognized service
$ sudo initctl status logstash
logstash stop/waiting
I am aware of the various ways to start a linux process, I am just pointing out that logstash is the only component which does not conform to the others. I am fine with whatever process, just saying that the phrase "one of these things does not belong, one of these things is not like the other..." comes to mind.
Managing server farms is difficult when a known good process, which has worked in previous versions, and works in all companion processes, does not follow the same guidelines.
My two cents. Sorry for the quasi rant.
5 Likes
I agree - the rest of the world uses services
So, in a standard Amazon Linux AMI (2016.09)
-
starting/stopping with
initctrlworks - well some process then runs BUT -
there is nothing e.g. in /var/log/logstash
-
even after a few minutes, the logstash process is #1 in
topusing >100% CPU
29230 logstash 20 0 3351m 274m 15m S 188.1 3.4 0:11.41 java
20919 elastics 20 0 10.0g 4.6g 247m S 9.0 59.3 19:07.62 javatime /usr/share/logstash/bin/logstash --help
just hangs - also running with sudo - but if you wait long enough, the result is
27.12user 0.63system 0:30.25elapsed 91%CPU (0avgtext+0avgdata 443276maxresident)k
16inputs+64outputs (0major+124605minor)pagefaults 0swaps
... a bit unusual output for time :-/
You can easily replicate this with a t2-micro or t2-small Amazon Linux instance.
(no cheating with CentOS or Vagrant - use the real thing - I will buy you a beer at {ON} for the costs - $0.026 per Hour)
When looking at the support metrics of elastic, there are "x" (i.e. not) in the column for Amazon Linux for Logstash - all the other products are green. So, it seems to be a known issue. Would be nice to know
- what this causes the issues
- if there is any plan/option to address this.
You're using a shared architecture. If Logstash is slow to start, try adding the extra entropy as linked above. Some users have seen 5 to 10 minute start times decrease to less than 20 seconds.
Hi,
And thanks for the support. I figure out that elasticsearch 5.x have a new feature ingest that can be enough to just parse logs. So no need to install Logstash.
I tried to follow the instruction in this blog post:
But there's only part-1, some typo errors and I can't make it work correctly. Can somebody confirm that the ingest is enough so I can dig more and try to make it work.
Hope it can help AWS users that just want a log server.
It may well be that Ingest node is sufficient for your needs. But Ingest is only a small subset of what Logstash does, so if you only need grok and GeoIP and such, then you're probably okay using it. If you need a feature that Ingest does not support, you'll need to use Logstash.
1 Like