Improve logstash starting time?

[asp]

Hi,

I know, it is not the hardware logstash is designed for, but hopefully you have some idea which might help me.

I am running a nextcloud instance an would like to visualize server's logs (nextcloud, apache, metrics) in kibana. I don't expect much events since it is only used by a few users.
It runs on an odroid xu4 (arm) with arch linux.
You can compare the performance with a fast atom processor, 2 GB memory.

I managed to get elasticsearch, kibana and beats running. startup takes a bit longer than on an ordinary x86 server, but it seems to work so far.
But with logstash I am struggling.

After starting it takes now 20 minutes to start - and it is not finished yet.
Any Idea how I can speed it up? I do not use any encryption between logstash and elasticsearch. I read, that there may be some entropy issues which might slow down the logstash start.

haveged is already running on the server.

cat /proc/sys/kernel/random/entropy_avail
gives back values about 3k.

the current logstash log shows:

[2017-10-06T12:59:37,349][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/data/elastic/logstash/current/modules/fb_apache/configuration"}
[2017-10-06T12:59:37,385][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/data/elastic/logstash/current/modules/netflow/configuration"}
[2017-10-06T13:20:06,097][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/data/elastic/logstash/current/modules/fb_apache/configuration"}
[2017-10-06T13:20:06,134][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/data/elastic/logstash/current/modules/netflow/configuration"}

Using elastic stack 5.6.2 with openjdk 8.

Any ideas for speeding it up is apreciated.
Regards, Andreas

[jakelandis]

Hi @asp,

You are correct about the entropy issues and startup times.

The issue is https://github.com/elastic/logstash/issues/6117 and was fixed here: https://github.com/elastic/logstash/commit/50cbaf4bf0205af278651208134467a4a23dcee8

...but we just noticed that changed did NOT make it into release packages (rpm, deb). That issue is here: https://github.com/elastic/logstash/issues/8427 (not fixed yet)

To fix this you can add the this to the bottom of your jvm.options.

# Entropy source for randomness
-Djava.security.egd=file:/dev/urandom

[asp]

i will try. I took the tar.gz file for manual installation btw.

Update: it is already set there.
might it be improved when setting it to /dev/random
?

Or any other things which might help to hunt the problem down? I doubt that is just the performance of the arm processor, because running a linux vm on atom processor is much faster and both cpu's have about the same performance.

[jakelandis]

I wonder if it isn't picking those jvm options then ? /dev/urandom should be non-blocking.

Try looking at the process, it should be an command line argument:
For example:

ps aux | grep logstash

 -Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -Xmx1g -Xms256m -Xss2048k -Djffi.boot.library.path=/usr/share/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/usr/share/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/share/logstash/vendor/jruby -Djruby.lib=/usr/share/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main /usr/share/logstash/lib/bootstrap/environment.rb logstash/runner.rb --path.settings /etc/logstash

[asp]

I tried to use oracle jdk instead of openjdk.
Now it starts in less than a minute.