Unable to start logstash

Mike_Reprogle · September 18, 2024, 4:20am

I have seen many other posts that seem similar, but come down to not having a config file set. I have a couple of .conf files located in /etc/logstash/conf.d, as I am trying to set up an Azure VM with logstash for SaaS logs.

I am not seeing much of anything to go off of in these logs, and even uninstalled/reinstalled logstash using the directions on the official page: Installing Logstash | Logstash Reference [8.15] | Elastic

I've tried multiple changes, and it always just ends up failing with the same message:

[2024-09-18T03:56:16,429][INFO ][logstash.runner          ] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2024-09-18T03:56:16,448][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.15.1", "jruby.version"=>"jruby 9.4.8.0 (3.1.4) 2024-07-02 4d41e55a67 OpenJDK 64-Bit Server VM 21.0.4+7-LTS on 21.0.4+7-LTS +indy +jit [x86_64-linux]"}
[2024-09-18T03:56:16,460][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Dlogstash.jackson.stream-read-constraints.max-string-length=200000000, -Dlogstash.jackson.stream-read-constraints.max-number-length=10000, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED, -Dio.netty.allocator.maxOrder=11]
[2024-09-18T03:56:16,470][INFO ][logstash.runner          ] Jackson default value override `logstash.jackson.stream-read-constraints.max-string-length` configured to `200000000`
[2024-09-18T03:56:16,470][INFO ][logstash.runner          ] Jackson default value override `logstash.jackson.stream-read-constraints.max-number-length` configured to `10000`
[2024-09-18T03:56:17,250][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
	at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:921) ~[jruby.jar:?]
	at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:880) ~[jruby.jar:?]
	at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:90) ~[?:?]

My pipelines.yml file looks like the following:

- pipeline.id: main
path.config: "/etc/logstash/conf.d/*.conf"
pipeline.ecs_compatibility: disabled

I even changed the log level to debug, and I am just not familiar enough with logstash to see anything glaring.

[2024-09-18T04:15:37,697][INFO ][logstash.runner          ] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2024-09-18T04:15:37,710][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.15.1", "jruby.version"=>"jruby 9.4.8.0 (3.1.4) 2024-07-02 4d41e55a67 OpenJDK 64-Bit Server VM 21.0.4+7-LTS on 21.0.4+7-LTS +indy +jit [x86_64-linux]"}
[2024-09-18T04:15:37,729][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Dlogstash.jackson.stream-read-constraints.max-string-length=200000000, -Dlogstash.jackson.stream-read-constraints.max-number-length=10000, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED, -Dio.netty.allocator.maxOrder=11]
[2024-09-18T04:15:37,730][DEBUG][logstash.modules.scaffold] Found module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2024-09-18T04:15:37,737][DEBUG][logstash.plugins.registry] Adding plugin to the registry {:name=>"netflow", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x6b21a869 @directory="/usr/share/logstash/modules/netflow/configuration", @module_name="netflow", @kibana_version_parts=["6", "0", "0"]>}
[2024-09-18T04:15:37,739][DEBUG][logstash.modules.scaffold] Found module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2024-09-18T04:15:37,739][DEBUG][logstash.plugins.registry] Adding plugin to the registry {:name=>"fb_apache", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x6edbdbed @directory="/usr/share/logstash/modules/fb_apache/configuration", @module_name="fb_apache", @kibana_version_parts=["6", "0", "0"]>}
[2024-09-18T04:15:37,748][INFO ][logstash.runner          ] Jackson default value override `logstash.jackson.stream-read-constraints.max-string-length` configured to `200000000`
[2024-09-18T04:15:37,749][INFO ][logstash.runner          ] Jackson default value override `logstash.jackson.stream-read-constraints.max-number-length` configured to `10000`
[2024-09-18T04:15:37,817][DEBUG][logstash.runner          ] Setting global FieldReference escape style: none
[2024-09-18T04:15:38,438][DEBUG][logstash.runner          ] -------- Logstash Settings (* means modified) ---------
[2024-09-18T04:15:38,438][DEBUG][logstash.runner          ] allow_superuser: true
[2024-09-18T04:15:38,439][DEBUG][logstash.runner          ] node.name: "FWM-Azure-Syslog"
[2024-09-18T04:15:38,439][DEBUG][logstash.runner          ] *path.data: "/var/lib/logstash" (default: "/usr/share/logstash/data")
[2024-09-18T04:15:38,446][DEBUG][logstash.runner          ] modules.cli: #<Java::OrgLogstashUtil::ModulesSettingArray: []>
[2024-09-18T04:15:38,446][DEBUG][logstash.runner          ] modules: []
[2024-09-18T04:15:38,446][DEBUG][logstash.runner          ] modules_list: []
[2024-09-18T04:15:38,447][DEBUG][logstash.runner          ] modules_variable_list: []
[2024-09-18T04:15:38,447][DEBUG][logstash.runner          ] modules_setup: false
[2024-09-18T04:15:38,447][DEBUG][logstash.runner          ] config.test_and_exit: false
[2024-09-18T04:15:38,447][DEBUG][logstash.runner          ] config.reload.automatic: false
[2024-09-18T04:15:38,447][DEBUG][logstash.runner          ] config.reload.interval: #<Java::OrgLogstashUtil::TimeValue:0x45ee48e7>
[2024-09-18T04:15:38,447][DEBUG][logstash.runner          ] config.support_escapes: false
[2024-09-18T04:15:38,447][DEBUG][logstash.runner          ] config.field_reference.escape_style: "none"
[2024-09-18T04:15:38,448][DEBUG][logstash.runner          ] event_api.tags.illegal: "rename"
[2024-09-18T04:15:38,449][DEBUG][logstash.runner          ] metric.collect: true
[2024-09-18T04:15:38,449][DEBUG][logstash.runner          ] pipeline.id: "main"
[2024-09-18T04:15:38,449][DEBUG][logstash.runner          ] pipeline.system: false
[2024-09-18T04:15:38,449][DEBUG][logstash.runner          ] pipeline.workers: 2
[2024-09-18T04:15:38,449][DEBUG][logstash.runner          ] pipeline.batch.size: 125
[2024-09-18T04:15:38,457][DEBUG][logstash.runner          ] pipeline.batch.delay: 50
[2024-09-18T04:15:38,457][DEBUG][logstash.runner          ] pipeline.unsafe_shutdown: false
[2024-09-18T04:15:38,458][DEBUG][logstash.runner          ] pipeline.reloadable: true
[2024-09-18T04:15:38,458][DEBUG][logstash.runner          ] pipeline.plugin_classloaders: false
[2024-09-18T04:15:38,458][DEBUG][logstash.runner          ] pipeline.separate_logs: false
[2024-09-18T04:15:38,458][DEBUG][logstash.runner          ] pipeline.ordered: "auto"
[2024-09-18T04:15:38,459][DEBUG][logstash.runner          ] pipeline.ecs_compatibility: "v8"
[2024-09-18T04:15:38,459][DEBUG][logstash.runner          ] path.plugins: []
[2024-09-18T04:15:38,466][DEBUG][logstash.runner          ] config.debug: false
[2024-09-18T04:15:38,466][DEBUG][logstash.runner          ] *log.level: "debug" (default: "info")
[2024-09-18T04:15:38,466][DEBUG][logstash.runner          ] version: false
[2024-09-18T04:15:38,467][DEBUG][logstash.runner          ] help: false
[2024-09-18T04:15:38,467][DEBUG][logstash.runner          ] enable-local-plugin-development: false
[2024-09-18T04:15:38,467][DEBUG][logstash.runner          ] log.format: "plain"
[2024-09-18T04:15:38,467][DEBUG][logstash.runner          ] log.format.json.fix_duplicate_message_fields: false
[2024-09-18T04:15:38,467][DEBUG][logstash.runner          ] api.enabled: true
[2024-09-18T04:15:38,467][DEBUG][logstash.runner          ] api.http.host: "127.0.0.1"
[2024-09-18T04:15:38,467][DEBUG][logstash.runner          ] api.http.port: 9600..9700
[2024-09-18T04:15:38,467][DEBUG][logstash.runner          ] api.environment: "production"
[2024-09-18T04:15:38,467][DEBUG][logstash.runner          ] api.auth.type: "none"
[2024-09-18T04:15:38,468][DEBUG][logstash.runner          ] api.auth.basic.password_policy.mode: "WARN"
[2024-09-18T04:15:38,468][DEBUG][logstash.runner          ] api.auth.basic.password_policy.length.minimum: 8
[2024-09-18T04:15:38,468][DEBUG][logstash.runner          ] api.auth.basic.password_policy.include.upper: "REQUIRED"
[2024-09-18T04:15:38,468][DEBUG][logstash.runner          ] api.auth.basic.password_policy.include.lower: "REQUIRED"
[2024-09-18T04:15:38,468][DEBUG][logstash.runner          ] api.auth.basic.password_policy.include.digit: "REQUIRED"
[2024-09-18T04:15:38,468][DEBUG][logstash.runner          ] api.auth.basic.password_policy.include.symbol: "OPTIONAL"
[2024-09-18T04:15:38,468][DEBUG][logstash.runner          ] api.ssl.enabled: false
[2024-09-18T04:15:38,476][DEBUG][logstash.runner          ] api.ssl.supported_protocols: []
[2024-09-18T04:15:38,476][DEBUG][logstash.runner          ] queue.type: "memory"
[2024-09-18T04:15:38,477][DEBUG][logstash.runner          ] queue.drain: false
[2024-09-18T04:15:38,477][DEBUG][logstash.runner          ] queue.page_capacity: 67108864
[2024-09-18T04:15:38,509][DEBUG][logstash.runner          ] queue.max_bytes: 1073741824
[2024-09-18T04:15:38,516][DEBUG][logstash.runner          ] queue.max_events: 0
[2024-09-18T04:15:38,516][DEBUG][logstash.runner          ] queue.checkpoint.acks: 1024
[2024-09-18T04:15:38,517][DEBUG][logstash.runner          ] queue.checkpoint.writes: 1024
[2024-09-18T04:15:38,517][DEBUG][logstash.runner          ] queue.checkpoint.interval: 1000
[2024-09-18T04:15:38,517][DEBUG][logstash.runner          ] queue.checkpoint.retry: true
[2024-09-18T04:15:38,518][DEBUG][logstash.runner          ] dead_letter_queue.enable: false
[2024-09-18T04:15:38,518][DEBUG][logstash.runner          ] dead_letter_queue.max_bytes: 1073741824
[2024-09-18T04:15:38,518][DEBUG][logstash.runner          ] dead_letter_queue.flush_interval: 5000
[2024-09-18T04:15:38,518][DEBUG][logstash.runner          ] dead_letter_queue.storage_policy: "drop_newer"
[2024-09-18T04:15:38,518][DEBUG][logstash.runner          ] slowlog.threshold.warn: #<Java::OrgLogstashUtil::TimeValue:0x389b2c60>
[2024-09-18T04:15:38,519][DEBUG][logstash.runner          ] slowlog.threshold.info: #<Java::OrgLogstashUtil::TimeValue:0x796a1cd1>
[2024-09-18T04:15:38,526][DEBUG][logstash.runner          ] slowlog.threshold.debug: #<Java::OrgLogstashUtil::TimeValue:0x17acd4c5>
[2024-09-18T04:15:38,526][DEBUG][logstash.runner          ] slowlog.threshold.trace: #<Java::OrgLogstashUtil::TimeValue:0x70bfc7d5>
[2024-09-18T04:15:38,527][DEBUG][logstash.runner          ] keystore.classname: "org.logstash.secret.store.backend.JavaKeyStore"
[2024-09-18T04:15:38,527][DEBUG][logstash.runner          ] *keystore.file: "/etc/logstash/logstash.keystore" (default: "/usr/share/logstash/config/logstash.keystore")
[2024-09-18T04:15:38,527][DEBUG][logstash.runner          ] pipeline.buffer.type: "direct"
[2024-09-18T04:15:38,527][DEBUG][logstash.runner          ] *path.queue: "/var/lib/logstash/queue" (default: "/usr/share/logstash/data/queue")
[2024-09-18T04:15:38,528][DEBUG][logstash.runner          ] *path.dead_letter_queue: "/var/lib/logstash/dead_letter_queue" (default: "/usr/share/logstash/data/dead_letter_queue")
[2024-09-18T04:15:38,528][DEBUG][logstash.runner          ] *path.settings: "/etc/logstash" (default: "/usr/share/logstash/config")
[2024-09-18T04:15:38,528][DEBUG][logstash.runner          ] *path.logs: "/var/log/logstash" (default: "/usr/share/logstash/logs")
[2024-09-18T04:15:38,528][DEBUG][logstash.runner          ] xpack.monitoring.enabled: false
[2024-09-18T04:15:38,529][DEBUG][logstash.runner          ] xpack.monitoring.elasticsearch.hosts: ["http://localhost:9200"]
[2024-09-18T04:15:38,529][DEBUG][logstash.runner          ] xpack.monitoring.collection.interval: #<Java::OrgLogstashUtil::TimeValue:0x50ed6392>
[2024-09-18T04:15:38,536][DEBUG][logstash.runner          ] xpack.monitoring.collection.timeout_interval: #<Java::OrgLogstashUtil::TimeValue:0x431ac922>
[2024-09-18T04:15:38,536][DEBUG][logstash.runner          ] xpack.monitoring.elasticsearch.username: "logstash_system"
[2024-09-18T04:15:38,536][DEBUG][logstash.runner          ] xpack.monitoring.elasticsearch.ssl.verification_mode: "full"
[2024-09-18T04:15:38,536][DEBUG][logstash.runner          ] xpack.monitoring.elasticsearch.ssl.cipher_suites: []
[2024-09-18T04:15:38,537][DEBUG][logstash.runner          ] xpack.monitoring.elasticsearch.sniffing: false
[2024-09-18T04:15:38,537][DEBUG][logstash.runner          ] xpack.monitoring.collection.pipeline.details.enabled: true
[2024-09-18T04:15:38,537][DEBUG][logstash.runner          ] xpack.monitoring.collection.config.enabled: true
[2024-09-18T04:15:38,537][DEBUG][logstash.runner          ] monitoring.enabled: false
[2024-09-18T04:15:38,537][DEBUG][logstash.runner          ] monitoring.elasticsearch.hosts: ["http://localhost:9200"]
[2024-09-18T04:15:38,537][DEBUG][logstash.runner          ] monitoring.collection.interval: #<Java::OrgLogstashUtil::TimeValue:0x50e7ee70>
[2024-09-18T04:15:38,537][DEBUG][logstash.runner          ] monitoring.collection.timeout_interval: #<Java::OrgLogstashUtil::TimeValue:0x287e35df>
[2024-09-18T04:15:38,537][DEBUG][logstash.runner          ] monitoring.elasticsearch.username: "logstash_system"
[2024-09-18T04:15:38,537][DEBUG][logstash.runner          ] monitoring.elasticsearch.ssl.verification_mode: "full"
[2024-09-18T04:15:38,538][DEBUG][logstash.runner          ] monitoring.elasticsearch.ssl.cipher_suites: []
[2024-09-18T04:15:38,538][DEBUG][logstash.runner          ] monitoring.elasticsearch.sniffing: false
[2024-09-18T04:15:38,538][DEBUG][logstash.runner          ] monitoring.collection.pipeline.details.enabled: true
[2024-09-18T04:15:38,538][DEBUG][logstash.runner          ] monitoring.collection.config.enabled: true
[2024-09-18T04:15:38,538][DEBUG][logstash.runner          ] node.uuid: ""
[2024-09-18T04:15:38,538][DEBUG][logstash.runner          ] xpack.geoip.downloader.endpoint: "https://geoip.elastic.co/v1/database"
[2024-09-18T04:15:38,538][DEBUG][logstash.runner          ] xpack.geoip.downloader.poll.interval: #<Java::OrgLogstashUtil::TimeValue:0x11cb348c>
[2024-09-18T04:15:38,538][DEBUG][logstash.runner          ] xpack.geoip.downloader.enabled: true
[2024-09-18T04:15:38,549][DEBUG][logstash.runner          ] xpack.management.enabled: false
[2024-09-18T04:15:38,556][DEBUG][logstash.runner          ] xpack.management.logstash.poll_interval: #<Java::OrgLogstashUtil::TimeValue:0x4d6cdebb>
[2024-09-18T04:15:38,556][DEBUG][logstash.runner          ] xpack.management.pipeline.id: ["main"]
[2024-09-18T04:15:38,557][DEBUG][logstash.runner          ] xpack.management.elasticsearch.username: "logstash_system"
[2024-09-18T04:15:38,557][DEBUG][logstash.runner          ] xpack.management.elasticsearch.hosts: ["https://localhost:9200"]
[2024-09-18T04:15:38,557][DEBUG][logstash.runner          ] xpack.management.elasticsearch.ssl.cipher_suites: []
[2024-09-18T04:15:38,557][DEBUG][logstash.runner          ] xpack.management.elasticsearch.ssl.verification_mode: "full"
[2024-09-18T04:15:38,557][DEBUG][logstash.runner          ] xpack.management.elasticsearch.sniffing: false
[2024-09-18T04:15:38,557][DEBUG][logstash.runner          ] --------------- Logstash Settings -------------------
[2024-09-18T04:15:38,618][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
	at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:921) ~[jruby.jar:?]
	at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:880) ~[jruby.jar:?]
	at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:90) ~[?:?]

If anyone knows these logs better than I do (it wouldn't take too much :), please let me know what I am doing wrong here. Outside of wiping my VM and starting over, I am not sure what to do here.

Rios · September 18, 2024, 6:34am

Check the journal log, there should be more info:
journalctl -u logstash.service -n 100

NgDinhNamEtiis · September 18, 2024, 10:42am

Even fresh install, the problem still exist then, may i suggest you create a new VM and run on that.

leandrojmp · September 18, 2024, 12:44pm

How are you running Logstash? As a service with systemctl?

If you are running as a service check the system logs, /var/log/syslog or /var/log/message, the way logstash seems to be crashing will not get logged in logstash logs, but on the system log.

Also, do your pipelines.yml lookes exactly like this or it was some typo while copying and pasting? The indentation is completely wrong.

It should be something like this:

- pipeline.id: main
  path.config: "/etc/logstash/conf.d/*.conf"
  pipeline.ecs_compatibility: disabled

Mike_Reprogle · September 24, 2024, 2:35am

Thanks for the heads up! I fixed the pipelines.yml, as there might have been some funky formatting in there that was messing it up. After setting this correctly, it did load my .conf files and I am able to make sample files now!

Topic		Replies	Views
Logstash not starting Logstash	7	443	October 4, 2022
Logstash stopped processing because of an error Logstash	4	114	June 20, 2024
Trying to start logstash in Azure - error Logstash	4	273	July 6, 2017
Logstash 8.1.0-1 fails to start Logstash	1	305	April 15, 2022
Simple config file but fail to start Logstash	7	357	January 1, 2019

Unable to start logstash

Related topics