Why is my Logstash startup so slow?

G82EtHAjnY4U · March 15, 2019, 4:28pm

I'm running Logstash 6.6.1 in a Docker container in a cloud computing environment (Heroku). There's a limit on app boot time on Heroku, which is problematic for me because Logstash sometimes takes up to 2 minutes to start.

I think I have a pretty simple setup:

Input via tcp
Extract a JSON component from each log entry
Output to Elasticsearch and S3

Here's my config and a log of the startup process:

gist.github.com

https://gist.github.com/dpmccabe/34dad1bd093182e72299c61f13a73c4e

Dockerfile

FROM logstash:6.6.1

ADD ./logstash.conf .
ADD ./logstash.yml .
ADD ./template.json .

ENV LS_JAVA_OPTS="-Dls.cgroup.cpuacct.path.override=/ -Dls.cgroup.cpu.path.override=/ -Djava.security.egd=file:/dev/urandom $LS_JAVA_OPTS"

CMD logstash -f ./logstash.conf --path.settings=. --verbose --log.level=debug

logstash.conf

input {
  tcp {
    port => "${PORT}"
  }
}

filter {
  grok {
    match => { "message" => "%{SYSLOG5424PRI:pri}%{NUMBER:rfc_version} %{TIMESTAMP_ISO8601:timestamp} %{USERNAME:app} %{USERNAME:dyno} %{USERNAME:proc} - Event %{USERNAME:event_id}: %{GREEDYDATA:message}" }
    overwrite => ["message"]

This file has been truncated. show original

logstash.yml

xpack.monitoring.enabled: false
xpack.management.enabled: false

There are more than three files. show original

It's not one thing that seems to be taking a long time. That log is from the container running on Heroku, but it takes about as long when I run it locally. Is this kind of startup performance expected for Logstash or is there something wrong with my setup?

kharvey · March 15, 2019, 4:33pm

Nope, that's about right.

My assumption is that most of the time is spent spinning up the JVM. Once the JVM is up and running, the rest of the logstash startup is pretty quick (maybe another 5 to 10 seconds).

I have this same problem on all of my VM's, physical servers, and cloud servers. This happens on fresh installs as well.

G82EtHAjnY4U · March 15, 2019, 4:48pm

That's disheartening. Do you think running inside Docker makes things much slower? I do have the option to just compile and run natively.

kharvey · March 15, 2019, 5:02pm

I just tested on my servers and it took 1:41 to startup.
I have a pretty complex pipeline setup on mine, I am running around a dozen pipelines right now over multiple ports.

Honestly, I have not tried logstash in Docker yet. My environment does not warrant that kind of separation...yet.

system · April 12, 2019, 5:02pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.