Logstash stopped processing because of an error: (SystemExit) exit

Elastic Stack Logstash
1

Logstash has suddenly stopped collecting.
If you look at the log, you will see the following message

Logstash stopped processing because of an error: (SystemExit) exit

What does this mean?
I have checked that the config description is correct and the status is healthy.
What do I need to do?

Here is the log file and the results of cofig and status, just in case.

  • log
$ tail /var/log/logstash/logstash-plain.log
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.13.0.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.13.0.jar:?]
        at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:89) ~[?:?]
[2021-06-13T09:46:34,649][INFO ][logstash.runner          ] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2021-06-13T09:46:34,658][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.12.0", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.10+9 on 11.0.10+9 +indy +jit [linux-x86_64]"}
[2021-06-13T09:46:35,023][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.13.0.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.13.0.jar:?]
        at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:89) ~[?:?]
  • config
$ /usr/share/logstash/bin/logstash -f /etc/logstash/logstash-its.conf --config.reload.automatic --config.test_and_exit
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2021-06-13 09:49:45.892 [main] runner - Starting Logstash {"logstash.version"=>"7.12.0", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.10+9 on 11.0.10+9 +indy +jit [linux-x86_64]"}
[WARN ] 2021-06-13 09:49:46.323 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2021-06-13 09:49:48.666 [LogStash::Runner] Reflections - Reflections took 37 ms to scan 1 urls, producing 23 keys and 47 values
[WARN ] 2021-06-13 09:49:48.966 [LogStash::Runner] beats - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2021-06-13 09:49:49.111 [LogStash::Runner] elasticsearch - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2021-06-13 09:49:49.129 [LogStash::Runner] elasticsearch - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2021-06-13 09:49:49.141 [LogStash::Runner] elasticsearch - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2021-06-13 09:49:49.153 [LogStash::Runner] elasticsearch - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2021-06-13 09:49:49.166 [LogStash::Runner] elasticsearch - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2021-06-13 09:49:49.177 [LogStash::Runner] elasticsearch - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2021-06-13 09:49:49.188 [LogStash::Runner] elasticsearch - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2021-06-13 09:49:49.198 [LogStash::Runner] elasticsearch - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2021-06-13 09:49:49.209 [LogStash::Runner] elasticsearch - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2021-06-13 09:49:49.219 [LogStash::Runner] elasticsearch - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2021-06-13 09:49:49.228 [LogStash::Runner] elasticsearch - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2021-06-13 09:49:49.236 [LogStash::Runner] elasticsearch - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2021-06-13 09:49:49.244 [LogStash::Runner] elasticsearch - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2021-06-13 09:49:49.253 [LogStash::Runner] elasticsearch - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2021-06-13 09:49:49.263 [LogStash::Runner] elasticsearch - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2021-06-13 09:49:49.272 [LogStash::Runner] elasticsearch - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2021-06-13 09:49:49.280 [LogStash::Runner] elasticsearch - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2021-06-13 09:49:49.288 [LogStash::Runner] elasticsearch - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2021-06-13 09:49:49.296 [LogStash::Runner] elasticsearch - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2021-06-13 09:49:49.304 [LogStash::Runner] elasticsearch - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2021-06-13 09:49:49.311 [LogStash::Runner] elasticsearch - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
Configuration OK
[INFO ] 2021-06-13 09:49:49.390 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
  • status
$  systemctl status logstash
● logstash.service - logstash
   Loaded: loaded (/etc/systemd/system/logstash.service; disabled; vendor preset: disabled)
   Active: active (running) since 日 2021-06-13 09:53:35 JST; 6s ago
 Main PID: 15737 (java)
   CGroup: /system.slice/logstash.service
           mq15737 /usr/share/logstash/jdk/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile...

 6月 13 09:53:35 ITS-ELS-02 systemd[1]: Started logstash.
 6月 13 09:53:35 ITS-ELS-02 logstash[15737]: Using bundled JDK: /usr/share/logstash/jdk
 6月 13 09:53:35 ITS-ELS-02 logstash[15737]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
2

Do you have more lines in logstash logs with FATAL or ERROR level? If yes, please share them.

Also, look into your system log, normally /var/log/messages, you could have something in your system that is not allowing logstash to start, since you are using systemd, it will be logged there.

3

Thanks for the answer.

There are no more logs in logstash. The same log is shown repeatedly.

I am attaching the system log. I don't think there is much understandable log here either.

$ tail -n 40 /var/log/messages
Jun 13 19:27:36 ITS-ELS-02 logstash: bin/logstash -V [--log.level fatal|error|warn|info|debug|trace]
Jun 13 19:27:36 ITS-ELS-02 logstash: bin/logstash --help
Jun 13 19:27:36 ITS-ELS-02 logstash: [2021-06-13T19:27:36,539][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
Jun 13 19:27:36 ITS-ELS-02 logstash: org.jruby.exceptions.SystemExit: (SystemExit) exit
Jun 13 19:27:36 ITS-ELS-02 logstash: at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.13.0.jar:?]
Jun 13 19:27:36 ITS-ELS-02 logstash: at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.13.0.jar:?]
Jun 13 19:27:36 ITS-ELS-02 logstash: at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:89) ~[?:?]
Jun 13 19:27:36 ITS-ELS-02 systemd: logstash.service: main process exited, code=exited, status=1/FAILURE
Jun 13 19:27:36 ITS-ELS-02 systemd: Unit logstash.service entered failed state.
Jun 13 19:27:36 ITS-ELS-02 systemd: logstash.service failed.
Jun 13 19:27:37 ITS-ELS-02 systemd: logstash.service holdoff time over, scheduling restart.
Jun 13 19:27:37 ITS-ELS-02 systemd: Stopped logstash.
Jun 13 19:27:37 ITS-ELS-02 systemd: Started logstash.
Jun 13 19:27:37 ITS-ELS-02 logstash: Using bundled JDK: /usr/share/logstash/jdk
Jun 13 19:27:37 ITS-ELS-02 logstash: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Jun 13 19:27:49 ITS-ELS-02 logstash: Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
Jun 13 19:27:50 ITS-ELS-02 logstash: [2021-06-13T19:27:50,104][INFO ][logstash.runner          ] Log4j configuration path used is: /etc/logstash/log4j2.properties
Jun 13 19:27:50 ITS-ELS-02 logstash: [2021-06-13T19:27:50,112][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.12.0", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.10+9 on 11.0.10+9 +indy +jit [linux-x86_64]"}
Jun 13 19:27:50 ITS-ELS-02 logstash: ERROR: Pipelines YAML file is empty. Location: /etc/logstash/pipelines.yml
Jun 13 19:27:50 ITS-ELS-02 logstash: usage:
Jun 13 19:27:50 ITS-ELS-02 logstash: bin/logstash -f CONFIG_PATH [-t] [-r] [] [-w COUNT] [-l LOG]
Jun 13 19:27:50 ITS-ELS-02 logstash: bin/logstash --modules MODULE_NAME [-M "MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAME.VARIABLE_NAME=VALUE"] [-t] [-w COUNT] [-l LOG]
Jun 13 19:27:50 ITS-ELS-02 logstash: bin/logstash -e CONFIG_STR [-t] [--log.level fatal|error|warn|info|debug|trace] [-w COUNT] [-l LOG]
Jun 13 19:27:50 ITS-ELS-02 logstash: bin/logstash -i SHELL [--log.level fatal|error|warn|info|debug|trace]
Jun 13 19:27:50 ITS-ELS-02 logstash: bin/logstash -V [--log.level fatal|error|warn|info|debug|trace]
Jun 13 19:27:50 ITS-ELS-02 logstash: bin/logstash --help
Jun 13 19:27:50 ITS-ELS-02 logstash: [2021-06-13T19:27:50,480][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
Jun 13 19:27:50 ITS-ELS-02 logstash: org.jruby.exceptions.SystemExit: (SystemExit) exit
Jun 13 19:27:50 ITS-ELS-02 logstash: at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.13.0.jar:?]
Jun 13 19:27:50 ITS-ELS-02 logstash: at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.13.0.jar:?]
Jun 13 19:27:50 ITS-ELS-02 logstash: at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:89) ~[?:?]
Jun 13 19:27:50 ITS-ELS-02 systemd: logstash.service: main process exited, code=exited, status=1/FAILURE
Jun 13 19:27:50 ITS-ELS-02 systemd: Unit logstash.service entered failed state.
Jun 13 19:27:50 ITS-ELS-02 systemd: logstash.service failed.
Jun 13 19:27:50 ITS-ELS-02 systemd: logstash.service holdoff time over, scheduling restart.
Jun 13 19:27:50 ITS-ELS-02 systemd: Stopped logstash.
Jun 13 19:27:50 ITS-ELS-02 systemd: Started logstash.
Jun 13 19:27:50 ITS-ELS-02 logstash: Using bundled JDK: /usr/share/logstash/jdk
Jun 13 19:27:51 ITS-ELS-02 logstash: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Jun 13 19:27:53 ITS-ELS-02 metricbeat: 2021-06-13T19:27:53.779+0900#011INFO#011[monitoring]#011log/log.go:144#011Non-zero metrics in the last 30s#011{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":63140280,"time":{"ms":365}},"total":{"ticks":105508130,"time":{"ms":614},"value":105508130},"user":{"ticks":42367850,"time":{"ms":249}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":9},"info":{"ephemeral_id":"e5b63d9e-3614-4596-aec1-994ba485c6fb","uptime":{"ms":5360010084}},"memstats":{"gc_next":19266048,"memory_alloc":12065640,"memory_total":12288829414432,"rss":38907904},"runtime":{"goroutines":61}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"acked":45,"active":16,"batches":6,"total":45},"read":{"bytes":2380},"write":{"bytes":127650}},"pipeline":{"clients":10,"events":{"active":0,"published":45,"total":45},"queue":{"acked":45}}},"metricbeat":{"system":{"cpu":{"events":3,"success":3},"filesystem":{"events":2,"success":2},"fsstat":{"events":1,"success":1},"load":{"events":3,"success":3},"memory":{"events":3,"success":3},"network":{"events":9,"success":9},"process":{"events":18,"success":18},"process_summary":{"events":3,"success":3},"socket_summary":{"events":3,"success":3}}},"system":{"load":{"1":4.16,"15":4.28,"5":4.18,"norm":{"1":0.52,"15":0.535,"5":0.5225}}}}}}
4

Hmmm. Still I do not know the cause of my error.
I know that in most cases there is an error in the logstash configuration file, but as mentioned above, my configuration file is said to be OK.
Is there something I'm missing?
Please advise me of any concerns you may have.

5

Does logstash always require a pipeline configuration?

Does it work with just the conf configuration?

6

Hi,

What's the command line used to start logstash ?

7

Thank you for your comment.

I am using the following command

$ /usr/share/logstash/bin/logstash -f /etc/logstash/logstash-its.conf --config.reload.automatic

The logstash-its.conf section describes the process of sending logs received from filebeat to elasticsearch.

8

ERROR: Pipelines YAML file is empty. Location: /etc/logstash/pipelines.yml
here is the problem I think... check your pipeline.yml file

9

Thank you for your comment.

My pipeline.yml file is as follows.

# cat /etc/logstash/pipelines.yml
# This file is where you define your pipelines. You can define multiple.
# For more information on multiple pipelines, see the documentation:
#   https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html

#- pipeline.id: main
#  path.config: ""#"/etc/logstash/conf.d/*.conf"

However, I use the -f option when starting logstash.

10

Can you try this in your pipelines.yml file ? and pass it to logstash directly using the -f option ?

- pipeline.id: main
path.config: "/etc/logstash/logstash-its.conf"

/bin/logstash -f pipelines.yml

Also can you read the files as the user you are running logstash with ?

11

The following changes have been made.

$ cat /etc/logstash/pipelines.yml
# This file is where you define your pipelines. You can define multiple.
# For more information on multiple pipelines, see the documentation:
#   https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html

- pipeline.id: main
#  path.config: "/etc/logstash/conf.d/*.conf"
  path.config: "/etc/logstash/logstash-its.conf"

I stopped logstash once and ran it again, but it stopped with another error.

# systemctl stop logstash
# /usr/share/logstash/bin/logstash -f /etc/logstash/pipelines.yml --config.reload.automatic
Using bundled JDK: /usr/share/logstash/jdk

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2021-06-14 21:16:01.422 [main] runner - Starting Logstash {"logstash.version"=>"7.12.0", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.10+9 on 11.0.10+9 +indy +jit [linux-x86_64]"}
[WARN ] 2021-06-14 21:16:01.735 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[FATAL] 2021-06-14 21:16:01.744 [LogStash::Runner] runner - Logstash could not be started because there is already another instance using the configured data directory.  If you wish to run multiple instances, you must change the "path.data" setting.
[FATAL] 2021-06-14 21:16:01.746 [LogStash::Runner] Logstash - Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.13.0.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.13.0.jar:?]
        at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:89) ~[?:?]
12

"/etc/logstash/conf.d/*.conf"
remove ""# from path.config

13

Thank you for your comment.

If I remove #, the result is the same.
My conf.d is an empty directory.

14

runner - Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting
i think already another instance of logstash is working which is not stopped yet... so first of all diasble it.
systemctl diasable logstash
and afterwards run the command for logstash..

15

Hmmm. It seems that even disable doesn't disable it properly.

# systemctl disable logstash
# /usr/share/logstash/bin/logstash -f /etc/logstash/pipelines.yml --config.reload.automatic
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2021-06-14 21:40:26.470 [main] runner - Starting Logstash {"logstash.version"=>"7.12.0", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.10+9 on 11.0.10+9 +indy +jit [linux-x86_64]"}
[WARN ] 2021-06-14 21:40:26.870 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[FATAL] 2021-06-14 21:40:26.888 [LogStash::Runner] runner - Logstash could not be started because there is already another instance using the configured data directory.  If you wish to run multiple instances, you must change the "path.data" setting.
[FATAL] 2021-06-14 21:40:26.892 [LogStash::Runner] Logstash - Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.13.0.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.13.0.jar:?]
        at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:89) ~[?:?]
16

How are you running logstash?

Using the command line or using systemctl? If you are using the command line with the parameter -f, then systemctl will do nothing.

You will need to kill the process of this logstash.

First check if there is really a logstash process running with ps, then get the PID of this process and kill it.

If you want to use systemctl start/stop, then you will need to use the pipelines.yml pointing to your config file.

17

I started logstash with the following command.

# /usr/share/logstash/bin/logstash -f /etc/logstash/logstash-its.conf --config.reload.automatic

Or, since you are installing the rpm package, do you recommend the following command?

# systemctl start logstash

Note that the result of ps is as follows.

# ps
  PID TTY          TIME CMD
19830 pts/2    00:00:00 bash
20342 pts/2    00:00:00 npm
20352 pts/2    00:00:01 grunt
24163 pts/2    00:00:00 ps

Does logstash apply to any of these?

18

I can't find logstash in ps, but I am told that logstash is running.

# ps -e | grep logstash
# ps -e
  PID TTY          TIME CMD
    1 ?        00:11:49 systemd
    2 ?        00:00:20 kthreadd
    4 ?        00:00:00 kworker/0:0H
    6 ?        00:03:49 ksoftirqd/0
    7 ?        00:00:04 migration/0
    8 ?        00:00:00 rcu_bh
    9 ?        01:12:36 rcu_sched
   10 ?        00:00:00 lru-add-drain
   11 ?        00:00:13 watchdog/0
   12 ?        00:00:15 watchdog/1
   13 ?        00:00:12 migration/1
   14 ?        00:03:47 ksoftirqd/1
   16 ?        00:00:00 kworker/1:0H
   17 ?        00:00:15 watchdog/2
   18 ?        00:00:07 migration/2
   19 ?        00:03:01 ksoftirqd/2
   21 ?        00:00:00 kworker/2:0H
   22 ?        00:00:14 watchdog/3
   23 ?        00:00:16 migration/3
   24 ?        00:02:44 ksoftirqd/3
   26 ?        00:00:00 kworker/3:0H
   27 ?        00:00:17 watchdog/4
   28 ?        00:00:06 migration/4
   29 ?        00:03:03 ksoftirqd/4
   31 ?        00:00:00 kworker/4:0H
   32 ?        00:00:18 watchdog/5
   33 ?        00:00:05 migration/5
   34 ?        00:03:04 ksoftirqd/5
   36 ?        00:00:00 kworker/5:0H
   37 ?        00:00:16 watchdog/6
   38 ?        00:00:10 migration/6
   39 ?        00:02:15 ksoftirqd/6
   41 ?        00:00:00 kworker/6:0H
   42 ?        00:00:13 watchdog/7
   43 ?        00:00:04 migration/7
   44 ?        00:03:50 ksoftirqd/7
   46 ?        00:00:00 kworker/7:0H
   48 ?        00:00:00 kdevtmpfs
   49 ?        00:00:00 netns
   50 ?        00:00:02 khungtaskd
   51 ?        00:00:00 writeback
   52 ?        00:00:00 kintegrityd
   53 ?        00:00:00 bioset
   54 ?        00:00:00 bioset
   55 ?        00:00:00 bioset
   56 ?        00:00:00 kblockd
   57 ?        00:00:00 md
   58 ?        00:00:00 edac-poller
   59 ?        00:00:00 watchdogd
   65 ?        23:23:09 kswapd0
   66 ?        00:00:00 ksmd
   67 ?        00:00:13 khugepaged
   68 ?        00:00:00 crypto
   76 ?        00:00:00 kthrotld
   78 ?        00:00:00 kmpath_rdacd
   79 ?        00:00:00 kaluad
   81 ?        00:00:00 kpsmoused
   83 ?        00:00:00 ipv6_addrconf
   96 ?        00:00:00 deferwq
  135 ?        00:00:08 kauditd
  312 ?        00:00:00 scsi_eh_0
  313 ?        00:00:00 scsi_tmf_0
  314 ?        00:00:00 vmw_pvscsi_wq_0
  319 ?        00:00:00 nfit
  322 ?        00:00:00 ata_sff
  323 ?        00:00:00 scsi_eh_1
  324 ?        00:00:00 scsi_tmf_1
  325 ?        00:00:00 scsi_eh_2
  326 ?        00:00:00 scsi_tmf_2
  338 ?        00:00:00 scsi_eh_3
  339 ?        00:00:00 scsi_tmf_3
  340 ?        00:00:00 scsi_eh_4
  341 ?        00:00:00 scsi_tmf_4
  342 ?        00:00:00 scsi_eh_5
  343 ?        00:00:00 scsi_tmf_5
  344 ?        00:00:00 scsi_eh_6
  345 ?        00:00:00 scsi_tmf_6
  346 ?        00:00:00 scsi_eh_7
  348 ?        00:00:00 scsi_tmf_7
  349 ?        00:00:00 scsi_eh_8
  350 ?        00:00:00 scsi_tmf_8
  351 ?        00:00:00 scsi_eh_9
  352 ?        00:00:00 scsi_tmf_9
  353 ?        00:00:00 scsi_eh_10
  354 ?        00:00:00 scsi_tmf_10
  355 ?        00:00:00 scsi_eh_11
  356 ?        00:00:00 scsi_tmf_11
  357 ?        00:00:00 scsi_eh_12
  358 ?        00:00:00 scsi_tmf_12
  359 ?        00:00:00 scsi_eh_13
  360 ?        00:00:00 scsi_tmf_13
  361 ?        00:00:00 scsi_eh_14
  362 ?        00:00:00 scsi_tmf_14
  363 ?        00:00:00 scsi_eh_15
  364 ?        00:00:00 scsi_tmf_15
  365 ?        00:00:00 scsi_eh_16
  366 ?        00:00:00 scsi_tmf_16
  367 ?        00:00:00 scsi_eh_17
  368 ?        00:00:00 scsi_tmf_17
  369 ?        00:00:00 scsi_eh_18
  370 ?        00:00:00 scsi_tmf_18
  371 ?        00:00:00 scsi_eh_19
  372 ?        00:00:00 scsi_tmf_19
  373 ?        00:00:00 scsi_eh_20
  374 ?        00:00:00 scsi_tmf_20
  375 ?        00:00:00 scsi_eh_21
  376 ?        00:00:00 scsi_tmf_21
  377 ?        00:00:00 scsi_eh_22
  378 ?        00:00:00 scsi_tmf_22
  379 ?        00:00:00 scsi_eh_23
  380 ?        00:00:00 scsi_tmf_23
  381 ?        00:00:00 scsi_eh_24
  382 ?        00:00:00 scsi_tmf_24
  383 ?        00:00:00 scsi_eh_25
  384 ?        00:00:00 scsi_tmf_25
  385 ?        00:00:00 scsi_eh_26
  386 ?        00:00:00 scsi_tmf_26
  387 ?        00:00:00 scsi_eh_27
  388 ?        00:00:00 scsi_tmf_27
  389 ?        00:00:00 scsi_eh_28
  390 ?        00:00:00 scsi_tmf_28
  391 ?        00:00:00 scsi_eh_29
  392 ?        00:00:00 scsi_tmf_29
  393 ?        00:00:00 scsi_eh_30
  394 ?        00:00:00 scsi_tmf_30
  395 ?        00:00:00 scsi_eh_31
  396 ?        00:00:00 scsi_tmf_31
  397 ?        00:00:00 scsi_eh_32
  398 ?        00:00:00 scsi_tmf_32
  431 ?        00:01:41 irq/16-vmwgfx
  432 ?        00:00:00 ttm_swap
  467 ?        00:00:09 kworker/0:1H
  475 ?        00:00:00 bioset
  476 ?        00:00:00 xfsalloc
  477 ?        00:00:00 xfs_mru_cache
  478 ?        00:00:00 xfs-buf/sda1
  479 ?        00:00:00 xfs-data/sda1
  480 ?        00:00:00 xfs-conv/sda1
  481 ?        00:00:00 xfs-cil/sda1
  482 ?        00:00:00 xfs-reclaim/sda
  483 ?        00:00:00 xfs-log/sda1
  484 ?        00:00:00 xfs-eofblocks/s
  485 ?        00:12:44 xfsaild/sda1
  486 ?        00:00:07 kworker/2:1H
  567 ?        01:27:19 systemd-journal
  604 ?        00:00:00 systemd-udevd
  605 ?        00:00:00 lvmetad
  631 ?        00:00:07 kworker/5:1H
  711 ?        00:00:00 kdmflush
  712 ?        00:00:00 bioset
  713 ?        00:00:00 kdmflush
  728 ?        04:36:02 jbd2/dm-1-8
  729 ?        00:00:00 ext4-rsv-conver
  746 ?        00:00:40 auditd
  769 ?        00:02:11 dbus-daemon
  771 ?        00:00:06 polkitd
  772 ?        00:00:00 VGAuthService
  773 ?        01:01:28 vmtoolsd
  780 ?        00:00:57 systemd-logind
  789 ?        00:03:57 irqbalance
  794 ?        00:00:11 crond
  800 tty1     00:00:00 agetty
  802 ?        00:00:09 chronyd
  811 ?        00:00:16 kworker/1:1H
 1083 ?        00:09:09 tuned
 1085 ?        00:00:00 sshd
 1087 ?        00:26:33 rsyslogd
 1098 ?        00:00:00 nginx
 1292 ?        00:00:16 master
 1294 ?        00:00:05 qmgr
 1407 ?        00:00:07 kworker/6:1H
 1454 ?        00:00:07 kworker/4:1H
 1504 ?        00:00:07 kworker/3:1H
 8299 ?        00:00:00 bioset
17621 ?        00:00:11 kworker/7:1H
18897 ?        10:54:18 node
18974 ?        00:00:33 nginx
19626 ?        00:00:00 kworker/4:0
19664 ?        00:00:01 sshd
19796 pts/0    00:00:00 bash
19826 pts/0    00:00:00 script
19829 pts/0    00:00:00 script
19830 pts/2    00:00:00 bash
19834 ?        27-06:07:25 java
20032 ?        00:00:00 controller
20281 ?        1-05:55:38 metricbeat
20342 pts/2    00:00:00 npm
20352 pts/2    00:00:01 grunt
21504 ?        00:00:02 kworker/4:2
21622 ?        00:00:00 kworker/2:3
21623 ?        00:00:00 kworker/2:4
21633 ?        00:00:01 kworker/7:10
21768 ?        00:00:00 kworker/6:1
21931 ?        00:00:00 kworker/3:6
21934 ?        00:00:00 kworker/3:9
21958 ?        00:00:00 kworker/5:14
21959 ?        00:00:00 kworker/5:15
23427 ?        34-22:38:51 java
23670 ?        00:00:03 kworker/7:0
23954 ?        00:00:00 pickup
24069 ?        00:00:00 kworker/0:1
24125 ?        00:00:01 kworker/u16:2
24128 ?        00:00:00 kworker/1:0
24134 ?        00:00:00 kworker/0:2
24155 ?        00:00:00 kworker/u16:3
24166 ?        00:00:00 kworker/1:1
24171 ?        00:00:00 kworker/u16:0
24172 ?        00:00:00 kworker/u16:1
24307 ?        00:00:00 kworker/1:2
24308 ?        00:00:00 kworker/0:0
24316 pts/2    00:00:00 ps
31228 ?        00:00:07 kworker/6:4

# /usr/share/logstash/bin/logstash -f /etc/logstash/logstash-its.conf --config.reload.automatic
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2021-06-14 23:47:10.853 [main] runner - Starting Logstash {"logstash.version"=>"7.12.0", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.10+9 on 11.0.10+9 +indy +jit [linux-x86_64]"}
[WARN ] 2021-06-14 23:47:11.213 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[FATAL] 2021-06-14 23:47:11.221 [LogStash::Runner] runner - Logstash could not be started because there is already another instance using the configured data directory.  If you wish to run multiple instances, you must change the "path.data" setting.
[FATAL] 2021-06-14 23:47:11.223 [LogStash::Runner] Logstash - Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.13.0.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.13.0.jar:?]
        at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:89) ~[?:?]
19

The .lock file could still be present.

It seems that you really do not have another logstash running, you can confirm that with the following command:

ps -ef | grep logstash

If there is really not a logstash process running, remove the .lock file from the directory /var/lib/logstash.

Then configure your pipelines.yml file as following:

- pipeline.id: main
  path.config: "/etc/logstash/logstash-its.conf"

After that start logstash using systemctl with systemctl start logstash.

Check the logs for any errors.

1 Like
20

Thank you very much.

I was able to stop logstash and start it anew using the procedure you showed me.

# ps -ef | grep logstash
root     23427     1 55  4月13 ?      34-23:24:38 /usr/share/logstash/jdk/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit.threshold=0 -Djruby.regexp.interruptible=true -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -Dlog4j2.isThreadContextMapInheritable=true -cp /usr/share/logstash/logstash-core/lib/jars/animal-sniffer-annotations-1.14.jar:/usr/share/logstash/logstash-core/lib/jars/checker-compat-qual-2.0.0.jar:/usr/share/logstash/logstash-core/lib/jars/commons-codec-1.14.jar:/usr/share/logstash/logstash-core/lib/jars/commons-compiler-3.1.0.jar:/usr/share/logstash/logstash-core/lib/jars/commons-logging-1.2.jar:/usr/share/logstash/logstash-core/lib/jars/error_prone_annotations-2.1.3.jar:/usr/share/logstash/logstash-core/lib/jars/google-java-format-1.1.jar:/usr/share/logstash/logstash-core/lib/jars/gradle-license-report-0.7.1.jar:/usr/share/logstash/logstash-core/lib/jars/guava-24.1.1-jre.jar:/usr/share/logstash/logstash-core/lib/jars/j2objc-annotations-1.1.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-annotations-2.9.10.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-core-2.9.10.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-databind-2.9.10.8.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.10.jar:/usr/share/logstash/logstash-core/lib/jars/janino-3.1.0.jar:/usr/share/logstash/logstash-core/lib/jars/javassist-3.26.0-GA.jar:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.13.0.jar:/usr/share/logstash/logstash-core/lib/jars/jsr305-1.3.9.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-api-2.13.3.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-core-2.13.3.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-jcl-2.13.3.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-slf4j-impl-2.13.3.jar:/usr/share/logstash/logstash-core/lib/jars/logstash-core.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.commands-3.6.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.contenttype-3.4.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.expressions-3.4.300.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.filesystem-1.3.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.jobs-3.5.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.resources-3.7.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.runtime-3.7.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.app-1.3.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.common-3.6.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.preferences-3.4.1.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.registry-3.5.101.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.jdt.core-3.10.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.osgi-3.7.1.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.text-3.5.101.jar:/usr/share/logstash/logstash-core/lib/jars/reflections-0.9.11.jar:/usr/share/logstash/logstash-core/lib/jars/slf4j-api-1.7.25.jar org.logstash.Logstash -f /etc/logstash/logstash-its.conf --config.reload.automatic
root     25249 19830  0 02:29 pts/2    00:00:00 grep --color=auto logstash
# kill -9 23427
# ps -ef | grep logstash
root     25258 19830  0 02:29 pts/2    00:00:00 grep --color=auto logstash

# systemctl start logstash
# systemctl status logstash
● logstash.service - logstash
   Loaded: loaded (/etc/systemd/system/logstash.service; disabled; vendor preset: disabled)
   Active: active (running) since 火 2021-06-15 02:30:09 JST; 7s ago
 Main PID: 25268 (java)
   CGroup: /system.slice/logstash.service
           mq25268 /usr/share/logstash/jdk/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=...

 6月 15 02:30:09 ITS-ELS-02 systemd[1]: Started logstash.
 6月 15 02:30:09 ITS-ELS-02 logstash[25268]: Using bundled JDK: /usr/share/logstash/jdk
 6月 15 02:30:10 ITS-ELS-02 logstash[25268]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.