Unable to start logstash

I have seen many other posts that seem similar, but come down to not having a config file set. I have a couple of .conf files located in /etc/logstash/conf.d, as I am trying to set up an Azure VM with logstash for SaaS logs.

I am not seeing much of anything to go off of in these logs, and even uninstalled/reinstalled logstash using the directions on the official page: Installing Logstash | Logstash Reference [8.15] | Elastic

I've tried multiple changes, and it always just ends up failing with the same message:

[2024-09-18T03:56:16,429][INFO ][logstash.runner          ] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2024-09-18T03:56:16,448][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.15.1", "jruby.version"=>"jruby 9.4.8.0 (3.1.4) 2024-07-02 4d41e55a67 OpenJDK 64-Bit Server VM 21.0.4+7-LTS on 21.0.4+7-LTS +indy +jit [x86_64-linux]"}
[2024-09-18T03:56:16,460][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Dlogstash.jackson.stream-read-constraints.max-string-length=200000000, -Dlogstash.jackson.stream-read-constraints.max-number-length=10000, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED, -Dio.netty.allocator.maxOrder=11]
[2024-09-18T03:56:16,470][INFO ][logstash.runner          ] Jackson default value override `logstash.jackson.stream-read-constraints.max-string-length` configured to `200000000`
[2024-09-18T03:56:16,470][INFO ][logstash.runner          ] Jackson default value override `logstash.jackson.stream-read-constraints.max-number-length` configured to `10000`
[2024-09-18T03:56:17,250][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
	at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:921) ~[jruby.jar:?]
	at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:880) ~[jruby.jar:?]
	at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:90) ~[?:?]

My pipelines.yml file looks like the following:

- pipeline.id: main
path.config: "/etc/logstash/conf.d/*.conf"
pipeline.ecs_compatibility: disabled

I even changed the log level to debug, and I am just not familiar enough with logstash to see anything glaring.

[2024-09-18T04:15:37,697][INFO ][logstash.runner          ] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2024-09-18T04:15:37,710][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.15.1", "jruby.version"=>"jruby 9.4.8.0 (3.1.4) 2024-07-02 4d41e55a67 OpenJDK 64-Bit Server VM 21.0.4+7-LTS on 21.0.4+7-LTS +indy +jit [x86_64-linux]"}
[2024-09-18T04:15:37,729][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Dlogstash.jackson.stream-read-constraints.max-string-length=200000000, -Dlogstash.jackson.stream-read-constraints.max-number-length=10000, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED, -Dio.netty.allocator.maxOrder=11]
[2024-09-18T04:15:37,730][DEBUG][logstash.modules.scaffold] Found module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2024-09-18T04:15:37,737][DEBUG][logstash.plugins.registry] Adding plugin to the registry {:name=>"netflow", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x6b21a869 @directory="/usr/share/logstash/modules/netflow/configuration", @module_name="netflow", @kibana_version_parts=["6", "0", "0"]>}
[2024-09-18T04:15:37,739][DEBUG][logstash.modules.scaffold] Found module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2024-09-18T04:15:37,739][DEBUG][logstash.plugins.registry] Adding plugin to the registry {:name=>"fb_apache", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x6edbdbed @directory="/usr/share/logstash/modules/fb_apache/configuration", @module_name="fb_apache", @kibana_version_parts=["6", "0", "0"]>}
[2024-09-18T04:15:37,748][INFO ][logstash.runner          ] Jackson default value override `logstash.jackson.stream-read-constraints.max-string-length` configured to `200000000`
[2024-09-18T04:15:37,749][INFO ][logstash.runner          ] Jackson default value override `logstash.jackson.stream-read-constraints.max-number-length` configured to `10000`
[2024-09-18T04:15:37,817][DEBUG][logstash.runner          ] Setting global FieldReference escape style: none
[2024-09-18T04:15:38,438][DEBUG][logstash.runner          ] -------- Logstash Settings (* means modified) ---------
[2024-09-18T04:15:38,438][DEBUG][logstash.runner          ] allow_superuser: true
[2024-09-18T04:15:38,439][DEBUG][logstash.runner          ] node.name: "FWM-Azure-Syslog"
[2024-09-18T04:15:38,439][DEBUG][logstash.runner          ] *path.data: "/var/lib/logstash" (default: "/usr/share/logstash/data")
[2024-09-18T04:15:38,446][DEBUG][logstash.runner          ] modules.cli: #<Java::OrgLogstashUtil::ModulesSettingArray: []>
[2024-09-18T04:15:38,446][DEBUG][logstash.runner          ] modules: []
[2024-09-18T04:15:38,446][DEBUG][logstash.runner          ] modules_list: []
[2024-09-18T04:15:38,447][DEBUG][logstash.runner          ] modules_variable_list: []
[2024-09-18T04:15:38,447][DEBUG][logstash.runner          ] modules_setup: false
[2024-09-18T04:15:38,447][DEBUG][logstash.runner          ] config.test_and_exit: false
[2024-09-18T04:15:38,447][DEBUG][logstash.runner          ] config.reload.automatic: false
[2024-09-18T04:15:38,447][DEBUG][logstash.runner          ] config.reload.interval: #<Java::OrgLogstashUtil::TimeValue:0x45ee48e7>
[2024-09-18T04:15:38,447][DEBUG][logstash.runner          ] config.support_escapes: false
[2024-09-18T04:15:38,447][DEBUG][logstash.runner          ] config.field_reference.escape_style: "none"
[2024-09-18T04:15:38,448][DEBUG][logstash.runner          ] event_api.tags.illegal: "rename"
[2024-09-18T04:15:38,449][DEBUG][logstash.runner          ] metric.collect: true
[2024-09-18T04:15:38,449][DEBUG][logstash.runner          ] pipeline.id: "main"
[2024-09-18T04:15:38,449][DEBUG][logstash.runner          ] pipeline.system: false
[2024-09-18T04:15:38,449][DEBUG][logstash.runner          ] pipeline.workers: 2
[2024-09-18T04:15:38,449][DEBUG][logstash.runner          ] pipeline.batch.size: 125
[2024-09-18T04:15:38,457][DEBUG][logstash.runner          ] pipeline.batch.delay: 50
[2024-09-18T04:15:38,457][DEBUG][logstash.runner          ] pipeline.unsafe_shutdown: false
[2024-09-18T04:15:38,458][DEBUG][logstash.runner          ] pipeline.reloadable: true
[2024-09-18T04:15:38,458][DEBUG][logstash.runner          ] pipeline.plugin_classloaders: false
[2024-09-18T04:15:38,458][DEBUG][logstash.runner          ] pipeline.separate_logs: false
[2024-09-18T04:15:38,458][DEBUG][logstash.runner          ] pipeline.ordered: "auto"
[2024-09-18T04:15:38,459][DEBUG][logstash.runner          ] pipeline.ecs_compatibility: "v8"
[2024-09-18T04:15:38,459][DEBUG][logstash.runner          ] path.plugins: []
[2024-09-18T04:15:38,466][DEBUG][logstash.runner          ] config.debug: false
[2024-09-18T04:15:38,466][DEBUG][logstash.runner          ] *log.level: "debug" (default: "info")
[2024-09-18T04:15:38,466][DEBUG][logstash.runner          ] version: false
[2024-09-18T04:15:38,467][DEBUG][logstash.runner          ] help: false
[2024-09-18T04:15:38,467][DEBUG][logstash.runner          ] enable-local-plugin-development: false
[2024-09-18T04:15:38,467][DEBUG][logstash.runner          ] log.format: "plain"
[2024-09-18T04:15:38,467][DEBUG][logstash.runner          ] log.format.json.fix_duplicate_message_fields: false
[2024-09-18T04:15:38,467][DEBUG][logstash.runner          ] api.enabled: true
[2024-09-18T04:15:38,467][DEBUG][logstash.runner          ] api.http.host: "127.0.0.1"
[2024-09-18T04:15:38,467][DEBUG][logstash.runner          ] api.http.port: 9600..9700
[2024-09-18T04:15:38,467][DEBUG][logstash.runner          ] api.environment: "production"
[2024-09-18T04:15:38,467][DEBUG][logstash.runner          ] api.auth.type: "none"
[2024-09-18T04:15:38,468][DEBUG][logstash.runner          ] api.auth.basic.password_policy.mode: "WARN"
[2024-09-18T04:15:38,468][DEBUG][logstash.runner          ] api.auth.basic.password_policy.length.minimum: 8
[2024-09-18T04:15:38,468][DEBUG][logstash.runner          ] api.auth.basic.password_policy.include.upper: "REQUIRED"
[2024-09-18T04:15:38,468][DEBUG][logstash.runner          ] api.auth.basic.password_policy.include.lower: "REQUIRED"
[2024-09-18T04:15:38,468][DEBUG][logstash.runner          ] api.auth.basic.password_policy.include.digit: "REQUIRED"
[2024-09-18T04:15:38,468][DEBUG][logstash.runner          ] api.auth.basic.password_policy.include.symbol: "OPTIONAL"
[2024-09-18T04:15:38,468][DEBUG][logstash.runner          ] api.ssl.enabled: false
[2024-09-18T04:15:38,476][DEBUG][logstash.runner          ] api.ssl.supported_protocols: []
[2024-09-18T04:15:38,476][DEBUG][logstash.runner          ] queue.type: "memory"
[2024-09-18T04:15:38,477][DEBUG][logstash.runner          ] queue.drain: false
[2024-09-18T04:15:38,477][DEBUG][logstash.runner          ] queue.page_capacity: 67108864
[2024-09-18T04:15:38,509][DEBUG][logstash.runner          ] queue.max_bytes: 1073741824
[2024-09-18T04:15:38,516][DEBUG][logstash.runner          ] queue.max_events: 0
[2024-09-18T04:15:38,516][DEBUG][logstash.runner          ] queue.checkpoint.acks: 1024
[2024-09-18T04:15:38,517][DEBUG][logstash.runner          ] queue.checkpoint.writes: 1024
[2024-09-18T04:15:38,517][DEBUG][logstash.runner          ] queue.checkpoint.interval: 1000
[2024-09-18T04:15:38,517][DEBUG][logstash.runner          ] queue.checkpoint.retry: true
[2024-09-18T04:15:38,518][DEBUG][logstash.runner          ] dead_letter_queue.enable: false
[2024-09-18T04:15:38,518][DEBUG][logstash.runner          ] dead_letter_queue.max_bytes: 1073741824
[2024-09-18T04:15:38,518][DEBUG][logstash.runner          ] dead_letter_queue.flush_interval: 5000
[2024-09-18T04:15:38,518][DEBUG][logstash.runner          ] dead_letter_queue.storage_policy: "drop_newer"
[2024-09-18T04:15:38,518][DEBUG][logstash.runner          ] slowlog.threshold.warn: #<Java::OrgLogstashUtil::TimeValue:0x389b2c60>
[2024-09-18T04:15:38,519][DEBUG][logstash.runner          ] slowlog.threshold.info: #<Java::OrgLogstashUtil::TimeValue:0x796a1cd1>
[2024-09-18T04:15:38,526][DEBUG][logstash.runner          ] slowlog.threshold.debug: #<Java::OrgLogstashUtil::TimeValue:0x17acd4c5>
[2024-09-18T04:15:38,526][DEBUG][logstash.runner          ] slowlog.threshold.trace: #<Java::OrgLogstashUtil::TimeValue:0x70bfc7d5>
[2024-09-18T04:15:38,527][DEBUG][logstash.runner          ] keystore.classname: "org.logstash.secret.store.backend.JavaKeyStore"
[2024-09-18T04:15:38,527][DEBUG][logstash.runner          ] *keystore.file: "/etc/logstash/logstash.keystore" (default: "/usr/share/logstash/config/logstash.keystore")
[2024-09-18T04:15:38,527][DEBUG][logstash.runner          ] pipeline.buffer.type: "direct"
[2024-09-18T04:15:38,527][DEBUG][logstash.runner          ] *path.queue: "/var/lib/logstash/queue" (default: "/usr/share/logstash/data/queue")
[2024-09-18T04:15:38,528][DEBUG][logstash.runner          ] *path.dead_letter_queue: "/var/lib/logstash/dead_letter_queue" (default: "/usr/share/logstash/data/dead_letter_queue")
[2024-09-18T04:15:38,528][DEBUG][logstash.runner          ] *path.settings: "/etc/logstash" (default: "/usr/share/logstash/config")
[2024-09-18T04:15:38,528][DEBUG][logstash.runner          ] *path.logs: "/var/log/logstash" (default: "/usr/share/logstash/logs")
[2024-09-18T04:15:38,528][DEBUG][logstash.runner          ] xpack.monitoring.enabled: false
[2024-09-18T04:15:38,529][DEBUG][logstash.runner          ] xpack.monitoring.elasticsearch.hosts: ["http://localhost:9200"]
[2024-09-18T04:15:38,529][DEBUG][logstash.runner          ] xpack.monitoring.collection.interval: #<Java::OrgLogstashUtil::TimeValue:0x50ed6392>
[2024-09-18T04:15:38,536][DEBUG][logstash.runner          ] xpack.monitoring.collection.timeout_interval: #<Java::OrgLogstashUtil::TimeValue:0x431ac922>
[2024-09-18T04:15:38,536][DEBUG][logstash.runner          ] xpack.monitoring.elasticsearch.username: "logstash_system"
[2024-09-18T04:15:38,536][DEBUG][logstash.runner          ] xpack.monitoring.elasticsearch.ssl.verification_mode: "full"
[2024-09-18T04:15:38,536][DEBUG][logstash.runner          ] xpack.monitoring.elasticsearch.ssl.cipher_suites: []
[2024-09-18T04:15:38,537][DEBUG][logstash.runner          ] xpack.monitoring.elasticsearch.sniffing: false
[2024-09-18T04:15:38,537][DEBUG][logstash.runner          ] xpack.monitoring.collection.pipeline.details.enabled: true
[2024-09-18T04:15:38,537][DEBUG][logstash.runner          ] xpack.monitoring.collection.config.enabled: true
[2024-09-18T04:15:38,537][DEBUG][logstash.runner          ] monitoring.enabled: false
[2024-09-18T04:15:38,537][DEBUG][logstash.runner          ] monitoring.elasticsearch.hosts: ["http://localhost:9200"]
[2024-09-18T04:15:38,537][DEBUG][logstash.runner          ] monitoring.collection.interval: #<Java::OrgLogstashUtil::TimeValue:0x50e7ee70>
[2024-09-18T04:15:38,537][DEBUG][logstash.runner          ] monitoring.collection.timeout_interval: #<Java::OrgLogstashUtil::TimeValue:0x287e35df>
[2024-09-18T04:15:38,537][DEBUG][logstash.runner          ] monitoring.elasticsearch.username: "logstash_system"
[2024-09-18T04:15:38,537][DEBUG][logstash.runner          ] monitoring.elasticsearch.ssl.verification_mode: "full"
[2024-09-18T04:15:38,538][DEBUG][logstash.runner          ] monitoring.elasticsearch.ssl.cipher_suites: []
[2024-09-18T04:15:38,538][DEBUG][logstash.runner          ] monitoring.elasticsearch.sniffing: false
[2024-09-18T04:15:38,538][DEBUG][logstash.runner          ] monitoring.collection.pipeline.details.enabled: true
[2024-09-18T04:15:38,538][DEBUG][logstash.runner          ] monitoring.collection.config.enabled: true
[2024-09-18T04:15:38,538][DEBUG][logstash.runner          ] node.uuid: ""
[2024-09-18T04:15:38,538][DEBUG][logstash.runner          ] xpack.geoip.downloader.endpoint: "https://geoip.elastic.co/v1/database"
[2024-09-18T04:15:38,538][DEBUG][logstash.runner          ] xpack.geoip.downloader.poll.interval: #<Java::OrgLogstashUtil::TimeValue:0x11cb348c>
[2024-09-18T04:15:38,538][DEBUG][logstash.runner          ] xpack.geoip.downloader.enabled: true
[2024-09-18T04:15:38,549][DEBUG][logstash.runner          ] xpack.management.enabled: false
[2024-09-18T04:15:38,556][DEBUG][logstash.runner          ] xpack.management.logstash.poll_interval: #<Java::OrgLogstashUtil::TimeValue:0x4d6cdebb>
[2024-09-18T04:15:38,556][DEBUG][logstash.runner          ] xpack.management.pipeline.id: ["main"]
[2024-09-18T04:15:38,557][DEBUG][logstash.runner          ] xpack.management.elasticsearch.username: "logstash_system"
[2024-09-18T04:15:38,557][DEBUG][logstash.runner          ] xpack.management.elasticsearch.hosts: ["https://localhost:9200"]
[2024-09-18T04:15:38,557][DEBUG][logstash.runner          ] xpack.management.elasticsearch.ssl.cipher_suites: []
[2024-09-18T04:15:38,557][DEBUG][logstash.runner          ] xpack.management.elasticsearch.ssl.verification_mode: "full"
[2024-09-18T04:15:38,557][DEBUG][logstash.runner          ] xpack.management.elasticsearch.sniffing: false
[2024-09-18T04:15:38,557][DEBUG][logstash.runner          ] --------------- Logstash Settings -------------------
[2024-09-18T04:15:38,618][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
	at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:921) ~[jruby.jar:?]
	at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:880) ~[jruby.jar:?]
	at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:90) ~[?:?]

If anyone knows these logs better than I do (it wouldn't take too much :), please let me know what I am doing wrong here. Outside of wiping my VM and starting over, I am not sure what to do here.

Check the journal log, there should be more info:
journalctl -u logstash.service -n 100

Even fresh install, the problem still exist then, may i suggest you create a new VM and run on that.

How are you running Logstash? As a service with systemctl?

If you are running as a service check the system logs, /var/log/syslog or /var/log/message, the way logstash seems to be crashing will not get logged in logstash logs, but on the system log.

Also, do your pipelines.yml lookes exactly like this or it was some typo while copying and pasting? The indentation is completely wrong.

It should be something like this:

- pipeline.id: main
  path.config: "/etc/logstash/conf.d/*.conf"
  pipeline.ecs_compatibility: disabled
1 Like

Thanks for the heads up! I fixed the pipelines.yml, as there might have been some funky formatting in there that was messing it up. After setting this correctly, it did load my .conf files and I am able to make sample files now!