I have seen many other posts that seem similar, but come down to not having a config file set. I have a couple of .conf files located in /etc/logstash/conf.d, as I am trying to set up an Azure VM with logstash for SaaS logs.
I am not seeing much of anything to go off of in these logs, and even uninstalled/reinstalled logstash using the directions on the official page: Installing Logstash | Logstash Reference [8.15] | Elastic
I've tried multiple changes, and it always just ends up failing with the same message:
[2024-09-18T03:56:16,429][INFO ][logstash.runner ] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2024-09-18T03:56:16,448][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"8.15.1", "jruby.version"=>"jruby 9.4.8.0 (3.1.4) 2024-07-02 4d41e55a67 OpenJDK 64-Bit Server VM 21.0.4+7-LTS on 21.0.4+7-LTS +indy +jit [x86_64-linux]"}
[2024-09-18T03:56:16,460][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Dlogstash.jackson.stream-read-constraints.max-string-length=200000000, -Dlogstash.jackson.stream-read-constraints.max-number-length=10000, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED, -Dio.netty.allocator.maxOrder=11]
[2024-09-18T03:56:16,470][INFO ][logstash.runner ] Jackson default value override `logstash.jackson.stream-read-constraints.max-string-length` configured to `200000000`
[2024-09-18T03:56:16,470][INFO ][logstash.runner ] Jackson default value override `logstash.jackson.stream-read-constraints.max-number-length` configured to `10000`
[2024-09-18T03:56:17,250][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:921) ~[jruby.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:880) ~[jruby.jar:?]
at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:90) ~[?:?]
My pipelines.yml file looks like the following:
- pipeline.id: main
path.config: "/etc/logstash/conf.d/*.conf"
pipeline.ecs_compatibility: disabled
I even changed the log level to debug, and I am just not familiar enough with logstash to see anything glaring.
[2024-09-18T04:15:37,697][INFO ][logstash.runner ] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2024-09-18T04:15:37,710][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"8.15.1", "jruby.version"=>"jruby 9.4.8.0 (3.1.4) 2024-07-02 4d41e55a67 OpenJDK 64-Bit Server VM 21.0.4+7-LTS on 21.0.4+7-LTS +indy +jit [x86_64-linux]"}
[2024-09-18T04:15:37,729][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Dlogstash.jackson.stream-read-constraints.max-string-length=200000000, -Dlogstash.jackson.stream-read-constraints.max-number-length=10000, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED, -Dio.netty.allocator.maxOrder=11]
[2024-09-18T04:15:37,730][DEBUG][logstash.modules.scaffold] Found module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2024-09-18T04:15:37,737][DEBUG][logstash.plugins.registry] Adding plugin to the registry {:name=>"netflow", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x6b21a869 @directory="/usr/share/logstash/modules/netflow/configuration", @module_name="netflow", @kibana_version_parts=["6", "0", "0"]>}
[2024-09-18T04:15:37,739][DEBUG][logstash.modules.scaffold] Found module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2024-09-18T04:15:37,739][DEBUG][logstash.plugins.registry] Adding plugin to the registry {:name=>"fb_apache", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x6edbdbed @directory="/usr/share/logstash/modules/fb_apache/configuration", @module_name="fb_apache", @kibana_version_parts=["6", "0", "0"]>}
[2024-09-18T04:15:37,748][INFO ][logstash.runner ] Jackson default value override `logstash.jackson.stream-read-constraints.max-string-length` configured to `200000000`
[2024-09-18T04:15:37,749][INFO ][logstash.runner ] Jackson default value override `logstash.jackson.stream-read-constraints.max-number-length` configured to `10000`
[2024-09-18T04:15:37,817][DEBUG][logstash.runner ] Setting global FieldReference escape style: none
[2024-09-18T04:15:38,438][DEBUG][logstash.runner ] -------- Logstash Settings (* means modified) ---------
[2024-09-18T04:15:38,438][DEBUG][logstash.runner ] allow_superuser: true
[2024-09-18T04:15:38,439][DEBUG][logstash.runner ] node.name: "FWM-Azure-Syslog"
[2024-09-18T04:15:38,439][DEBUG][logstash.runner ] *path.data: "/var/lib/logstash" (default: "/usr/share/logstash/data")
[2024-09-18T04:15:38,446][DEBUG][logstash.runner ] modules.cli: #<Java::OrgLogstashUtil::ModulesSettingArray: []>
[2024-09-18T04:15:38,446][DEBUG][logstash.runner ] modules: []
[2024-09-18T04:15:38,446][DEBUG][logstash.runner ] modules_list: []
[2024-09-18T04:15:38,447][DEBUG][logstash.runner ] modules_variable_list: []
[2024-09-18T04:15:38,447][DEBUG][logstash.runner ] modules_setup: false
[2024-09-18T04:15:38,447][DEBUG][logstash.runner ] config.test_and_exit: false
[2024-09-18T04:15:38,447][DEBUG][logstash.runner ] config.reload.automatic: false
[2024-09-18T04:15:38,447][DEBUG][logstash.runner ] config.reload.interval: #<Java::OrgLogstashUtil::TimeValue:0x45ee48e7>
[2024-09-18T04:15:38,447][DEBUG][logstash.runner ] config.support_escapes: false
[2024-09-18T04:15:38,447][DEBUG][logstash.runner ] config.field_reference.escape_style: "none"
[2024-09-18T04:15:38,448][DEBUG][logstash.runner ] event_api.tags.illegal: "rename"
[2024-09-18T04:15:38,449][DEBUG][logstash.runner ] metric.collect: true
[2024-09-18T04:15:38,449][DEBUG][logstash.runner ] pipeline.id: "main"
[2024-09-18T04:15:38,449][DEBUG][logstash.runner ] pipeline.system: false
[2024-09-18T04:15:38,449][DEBUG][logstash.runner ] pipeline.workers: 2
[2024-09-18T04:15:38,449][DEBUG][logstash.runner ] pipeline.batch.size: 125
[2024-09-18T04:15:38,457][DEBUG][logstash.runner ] pipeline.batch.delay: 50
[2024-09-18T04:15:38,457][DEBUG][logstash.runner ] pipeline.unsafe_shutdown: false
[2024-09-18T04:15:38,458][DEBUG][logstash.runner ] pipeline.reloadable: true
[2024-09-18T04:15:38,458][DEBUG][logstash.runner ] pipeline.plugin_classloaders: false
[2024-09-18T04:15:38,458][DEBUG][logstash.runner ] pipeline.separate_logs: false
[2024-09-18T04:15:38,458][DEBUG][logstash.runner ] pipeline.ordered: "auto"
[2024-09-18T04:15:38,459][DEBUG][logstash.runner ] pipeline.ecs_compatibility: "v8"
[2024-09-18T04:15:38,459][DEBUG][logstash.runner ] path.plugins: []
[2024-09-18T04:15:38,466][DEBUG][logstash.runner ] config.debug: false
[2024-09-18T04:15:38,466][DEBUG][logstash.runner ] *log.level: "debug" (default: "info")
[2024-09-18T04:15:38,466][DEBUG][logstash.runner ] version: false
[2024-09-18T04:15:38,467][DEBUG][logstash.runner ] help: false
[2024-09-18T04:15:38,467][DEBUG][logstash.runner ] enable-local-plugin-development: false
[2024-09-18T04:15:38,467][DEBUG][logstash.runner ] log.format: "plain"
[2024-09-18T04:15:38,467][DEBUG][logstash.runner ] log.format.json.fix_duplicate_message_fields: false
[2024-09-18T04:15:38,467][DEBUG][logstash.runner ] api.enabled: true
[2024-09-18T04:15:38,467][DEBUG][logstash.runner ] api.http.host: "127.0.0.1"
[2024-09-18T04:15:38,467][DEBUG][logstash.runner ] api.http.port: 9600..9700
[2024-09-18T04:15:38,467][DEBUG][logstash.runner ] api.environment: "production"
[2024-09-18T04:15:38,467][DEBUG][logstash.runner ] api.auth.type: "none"
[2024-09-18T04:15:38,468][DEBUG][logstash.runner ] api.auth.basic.password_policy.mode: "WARN"
[2024-09-18T04:15:38,468][DEBUG][logstash.runner ] api.auth.basic.password_policy.length.minimum: 8
[2024-09-18T04:15:38,468][DEBUG][logstash.runner ] api.auth.basic.password_policy.include.upper: "REQUIRED"
[2024-09-18T04:15:38,468][DEBUG][logstash.runner ] api.auth.basic.password_policy.include.lower: "REQUIRED"
[2024-09-18T04:15:38,468][DEBUG][logstash.runner ] api.auth.basic.password_policy.include.digit: "REQUIRED"
[2024-09-18T04:15:38,468][DEBUG][logstash.runner ] api.auth.basic.password_policy.include.symbol: "OPTIONAL"
[2024-09-18T04:15:38,468][DEBUG][logstash.runner ] api.ssl.enabled: false
[2024-09-18T04:15:38,476][DEBUG][logstash.runner ] api.ssl.supported_protocols: []
[2024-09-18T04:15:38,476][DEBUG][logstash.runner ] queue.type: "memory"
[2024-09-18T04:15:38,477][DEBUG][logstash.runner ] queue.drain: false
[2024-09-18T04:15:38,477][DEBUG][logstash.runner ] queue.page_capacity: 67108864
[2024-09-18T04:15:38,509][DEBUG][logstash.runner ] queue.max_bytes: 1073741824
[2024-09-18T04:15:38,516][DEBUG][logstash.runner ] queue.max_events: 0
[2024-09-18T04:15:38,516][DEBUG][logstash.runner ] queue.checkpoint.acks: 1024
[2024-09-18T04:15:38,517][DEBUG][logstash.runner ] queue.checkpoint.writes: 1024
[2024-09-18T04:15:38,517][DEBUG][logstash.runner ] queue.checkpoint.interval: 1000
[2024-09-18T04:15:38,517][DEBUG][logstash.runner ] queue.checkpoint.retry: true
[2024-09-18T04:15:38,518][DEBUG][logstash.runner ] dead_letter_queue.enable: false
[2024-09-18T04:15:38,518][DEBUG][logstash.runner ] dead_letter_queue.max_bytes: 1073741824
[2024-09-18T04:15:38,518][DEBUG][logstash.runner ] dead_letter_queue.flush_interval: 5000
[2024-09-18T04:15:38,518][DEBUG][logstash.runner ] dead_letter_queue.storage_policy: "drop_newer"
[2024-09-18T04:15:38,518][DEBUG][logstash.runner ] slowlog.threshold.warn: #<Java::OrgLogstashUtil::TimeValue:0x389b2c60>
[2024-09-18T04:15:38,519][DEBUG][logstash.runner ] slowlog.threshold.info: #<Java::OrgLogstashUtil::TimeValue:0x796a1cd1>
[2024-09-18T04:15:38,526][DEBUG][logstash.runner ] slowlog.threshold.debug: #<Java::OrgLogstashUtil::TimeValue:0x17acd4c5>
[2024-09-18T04:15:38,526][DEBUG][logstash.runner ] slowlog.threshold.trace: #<Java::OrgLogstashUtil::TimeValue:0x70bfc7d5>
[2024-09-18T04:15:38,527][DEBUG][logstash.runner ] keystore.classname: "org.logstash.secret.store.backend.JavaKeyStore"
[2024-09-18T04:15:38,527][DEBUG][logstash.runner ] *keystore.file: "/etc/logstash/logstash.keystore" (default: "/usr/share/logstash/config/logstash.keystore")
[2024-09-18T04:15:38,527][DEBUG][logstash.runner ] pipeline.buffer.type: "direct"
[2024-09-18T04:15:38,527][DEBUG][logstash.runner ] *path.queue: "/var/lib/logstash/queue" (default: "/usr/share/logstash/data/queue")
[2024-09-18T04:15:38,528][DEBUG][logstash.runner ] *path.dead_letter_queue: "/var/lib/logstash/dead_letter_queue" (default: "/usr/share/logstash/data/dead_letter_queue")
[2024-09-18T04:15:38,528][DEBUG][logstash.runner ] *path.settings: "/etc/logstash" (default: "/usr/share/logstash/config")
[2024-09-18T04:15:38,528][DEBUG][logstash.runner ] *path.logs: "/var/log/logstash" (default: "/usr/share/logstash/logs")
[2024-09-18T04:15:38,528][DEBUG][logstash.runner ] xpack.monitoring.enabled: false
[2024-09-18T04:15:38,529][DEBUG][logstash.runner ] xpack.monitoring.elasticsearch.hosts: ["http://localhost:9200"]
[2024-09-18T04:15:38,529][DEBUG][logstash.runner ] xpack.monitoring.collection.interval: #<Java::OrgLogstashUtil::TimeValue:0x50ed6392>
[2024-09-18T04:15:38,536][DEBUG][logstash.runner ] xpack.monitoring.collection.timeout_interval: #<Java::OrgLogstashUtil::TimeValue:0x431ac922>
[2024-09-18T04:15:38,536][DEBUG][logstash.runner ] xpack.monitoring.elasticsearch.username: "logstash_system"
[2024-09-18T04:15:38,536][DEBUG][logstash.runner ] xpack.monitoring.elasticsearch.ssl.verification_mode: "full"
[2024-09-18T04:15:38,536][DEBUG][logstash.runner ] xpack.monitoring.elasticsearch.ssl.cipher_suites: []
[2024-09-18T04:15:38,537][DEBUG][logstash.runner ] xpack.monitoring.elasticsearch.sniffing: false
[2024-09-18T04:15:38,537][DEBUG][logstash.runner ] xpack.monitoring.collection.pipeline.details.enabled: true
[2024-09-18T04:15:38,537][DEBUG][logstash.runner ] xpack.monitoring.collection.config.enabled: true
[2024-09-18T04:15:38,537][DEBUG][logstash.runner ] monitoring.enabled: false
[2024-09-18T04:15:38,537][DEBUG][logstash.runner ] monitoring.elasticsearch.hosts: ["http://localhost:9200"]
[2024-09-18T04:15:38,537][DEBUG][logstash.runner ] monitoring.collection.interval: #<Java::OrgLogstashUtil::TimeValue:0x50e7ee70>
[2024-09-18T04:15:38,537][DEBUG][logstash.runner ] monitoring.collection.timeout_interval: #<Java::OrgLogstashUtil::TimeValue:0x287e35df>
[2024-09-18T04:15:38,537][DEBUG][logstash.runner ] monitoring.elasticsearch.username: "logstash_system"
[2024-09-18T04:15:38,537][DEBUG][logstash.runner ] monitoring.elasticsearch.ssl.verification_mode: "full"
[2024-09-18T04:15:38,538][DEBUG][logstash.runner ] monitoring.elasticsearch.ssl.cipher_suites: []
[2024-09-18T04:15:38,538][DEBUG][logstash.runner ] monitoring.elasticsearch.sniffing: false
[2024-09-18T04:15:38,538][DEBUG][logstash.runner ] monitoring.collection.pipeline.details.enabled: true
[2024-09-18T04:15:38,538][DEBUG][logstash.runner ] monitoring.collection.config.enabled: true
[2024-09-18T04:15:38,538][DEBUG][logstash.runner ] node.uuid: ""
[2024-09-18T04:15:38,538][DEBUG][logstash.runner ] xpack.geoip.downloader.endpoint: "https://geoip.elastic.co/v1/database"
[2024-09-18T04:15:38,538][DEBUG][logstash.runner ] xpack.geoip.downloader.poll.interval: #<Java::OrgLogstashUtil::TimeValue:0x11cb348c>
[2024-09-18T04:15:38,538][DEBUG][logstash.runner ] xpack.geoip.downloader.enabled: true
[2024-09-18T04:15:38,549][DEBUG][logstash.runner ] xpack.management.enabled: false
[2024-09-18T04:15:38,556][DEBUG][logstash.runner ] xpack.management.logstash.poll_interval: #<Java::OrgLogstashUtil::TimeValue:0x4d6cdebb>
[2024-09-18T04:15:38,556][DEBUG][logstash.runner ] xpack.management.pipeline.id: ["main"]
[2024-09-18T04:15:38,557][DEBUG][logstash.runner ] xpack.management.elasticsearch.username: "logstash_system"
[2024-09-18T04:15:38,557][DEBUG][logstash.runner ] xpack.management.elasticsearch.hosts: ["https://localhost:9200"]
[2024-09-18T04:15:38,557][DEBUG][logstash.runner ] xpack.management.elasticsearch.ssl.cipher_suites: []
[2024-09-18T04:15:38,557][DEBUG][logstash.runner ] xpack.management.elasticsearch.ssl.verification_mode: "full"
[2024-09-18T04:15:38,557][DEBUG][logstash.runner ] xpack.management.elasticsearch.sniffing: false
[2024-09-18T04:15:38,557][DEBUG][logstash.runner ] --------------- Logstash Settings -------------------
[2024-09-18T04:15:38,618][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:921) ~[jruby.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:880) ~[jruby.jar:?]
at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:90) ~[?:?]
If anyone knows these logs better than I do (it wouldn't take too much :), please let me know what I am doing wrong here. Outside of wiping my VM and starting over, I am not sure what to do here.