Default install of logstash 8.1 results in error

Louis_Bohm · March 28, 2022, 2:25pm

I am running Centos 7.9 and followed the directions on this web to install logstash. After doing a yum -y install logstash to get logstash 8.1 installed did the following:
Removed the logstash-sample.conf.
Ran the following command: /usr/share/logstash/bin/logstash --config.test_and_exit -f /etc/logstash --log.level=debug

This resulted in the following error:

[FATAL] 2022-03-28 10:14:49.499 [LogStash::Runner] runner - The given configuration is invalid. Reason: Expected one of [ \t\r\n], "#", "input", "filter", "output" at line 6, column 1 (byte 132) after
[FATAL] 2022-03-28 10:14:49.502 [LogStash::Runner] Logstash - Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit

Not really sure how I can get this error since I have a totally default install with no config. Here is the full screen dump.

Logstash error on base install 
[root@gp2dc1sechids01 logstash]# /usr/share/logstash/bin/logstash --config.test_and_exit -f /etc/logstash --log.level=debug
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2022-03-28 10:14:47.983 [main] runner - Starting Logstash {"logstash.version"=>"8.1.1", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.14.1+1 on 11.0.14.1+1 +indy +jit [linux-x86_64]"}
[INFO ] 2022-03-28 10:14:47.992 [main] runner - JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[DEBUG] 2022-03-28 10:14:48.001 [main] scaffold - Found module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[DEBUG] 2022-03-28 10:14:48.002 [main] registry - Adding plugin to the registry {:name=>"fb_apache", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x27619d96 @directory="/usr/share/logstash/modules/fb_apache/configuration", @module_name="fb_apache", @kibana_version_parts=["6", "0", "0"]>}
[DEBUG] 2022-03-28 10:14:48.004 [main] scaffold - Found module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[DEBUG] 2022-03-28 10:14:48.004 [main] registry - Adding plugin to the registry {:name=>"netflow", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x28d5abce @directory="/usr/share/logstash/modules/netflow/configuration", @module_name="netflow", @kibana_version_parts=["6", "0", "0"]>}
[DEBUG] 2022-03-28 10:14:48.483 [LogStash::Runner] runner - -------- Logstash Settings (* means modified) ---------
[DEBUG] 2022-03-28 10:14:48.483 [LogStash::Runner] runner - node.name: "gp2dc1sechids01.tdsops.net"
[DEBUG] 2022-03-28 10:14:48.484 [LogStash::Runner] runner - *path.config: "/etc/logstash"
[DEBUG] 2022-03-28 10:14:48.484 [LogStash::Runner] runner - path.data: "/usr/share/logstash/data"
[DEBUG] 2022-03-28 10:14:48.484 [LogStash::Runner] runner - modules.cli: <Java::OrgLogstashUtil::ModulesSettingArray:1 []>
[DEBUG] 2022-03-28 10:14:48.484 [LogStash::Runner] runner - modules: []
[DEBUG] 2022-03-28 10:14:48.484 [LogStash::Runner] runner - modules_list: []
[DEBUG] 2022-03-28 10:14:48.484 [LogStash::Runner] runner - modules_variable_list: []
[DEBUG] 2022-03-28 10:14:48.484 [LogStash::Runner] runner - modules_setup: false
[DEBUG] 2022-03-28 10:14:48.484 [LogStash::Runner] runner - *config.test_and_exit: true (default: false)
[DEBUG] 2022-03-28 10:14:48.484 [LogStash::Runner] runner - config.reload.automatic: false
[DEBUG] 2022-03-28 10:14:48.484 [LogStash::Runner] runner - config.reload.interval: #<Java::OrgLogstashUtil::TimeValue:0x6894a9f1>
[DEBUG] 2022-03-28 10:14:48.484 [LogStash::Runner] runner - config.support_escapes: false
[DEBUG] 2022-03-28 10:14:48.485 [LogStash::Runner] runner - metric.collect: true
[DEBUG] 2022-03-28 10:14:48.485 [LogStash::Runner] runner - pipeline.id: "main"
[DEBUG] 2022-03-28 10:14:48.485 [LogStash::Runner] runner - pipeline.system: false
[DEBUG] 2022-03-28 10:14:48.486 [LogStash::Runner] runner - pipeline.workers: 4
[DEBUG] 2022-03-28 10:14:48.486 [LogStash::Runner] runner - pipeline.batch.size: 125
[DEBUG] 2022-03-28 10:14:48.486 [LogStash::Runner] runner - pipeline.batch.delay: 50
[DEBUG] 2022-03-28 10:14:48.486 [LogStash::Runner] runner - pipeline.unsafe_shutdown: false
[DEBUG] 2022-03-28 10:14:48.487 [LogStash::Runner] runner - pipeline.reloadable: true
[DEBUG] 2022-03-28 10:14:48.487 [LogStash::Runner] runner - pipeline.plugin_classloaders: false
[DEBUG] 2022-03-28 10:14:48.487 [LogStash::Runner] runner - pipeline.separate_logs: false
[DEBUG] 2022-03-28 10:14:48.487 [LogStash::Runner] runner - pipeline.ordered: "auto"
[DEBUG] 2022-03-28 10:14:48.487 [LogStash::Runner] runner - pipeline.ecs_compatibility: "v8"
[DEBUG] 2022-03-28 10:14:48.487 [LogStash::Runner] runner - path.plugins: []
[DEBUG] 2022-03-28 10:14:48.487 [LogStash::Runner] runner - config.debug: false
[DEBUG] 2022-03-28 10:14:48.487 [LogStash::Runner] runner - *log.level: "debug" (default: "info")
[DEBUG] 2022-03-28 10:14:48.487 [LogStash::Runner] runner - version: false
[DEBUG] 2022-03-28 10:14:48.487 [LogStash::Runner] runner - help: false
[DEBUG] 2022-03-28 10:14:48.488 [LogStash::Runner] runner - enable-local-plugin-development: false
[DEBUG] 2022-03-28 10:14:48.488 [LogStash::Runner] runner - log.format: "plain"
[DEBUG] 2022-03-28 10:14:48.488 [LogStash::Runner] runner - api.enabled: true
[DEBUG] 2022-03-28 10:14:48.488 [LogStash::Runner] runner - api.http.host: "127.0.0.1"
[DEBUG] 2022-03-28 10:14:48.488 [LogStash::Runner] runner - api.http.port: 9600..9700
[DEBUG] 2022-03-28 10:14:48.488 [LogStash::Runner] runner - api.environment: "production"
[DEBUG] 2022-03-28 10:14:48.488 [LogStash::Runner] runner - api.auth.type: "none"
[DEBUG] 2022-03-28 10:14:48.488 [LogStash::Runner] runner - api.ssl.enabled: false
[DEBUG] 2022-03-28 10:14:48.488 [LogStash::Runner] runner - queue.type: "memory"
[DEBUG] 2022-03-28 10:14:48.488 [LogStash::Runner] runner - queue.drain: false
[DEBUG] 2022-03-28 10:14:48.488 [LogStash::Runner] runner - queue.page_capacity: 67108864
[DEBUG] 2022-03-28 10:14:48.488 [LogStash::Runner] runner - queue.max_bytes: 1073741824
[DEBUG] 2022-03-28 10:14:48.489 [LogStash::Runner] runner - queue.max_events: 0
[DEBUG] 2022-03-28 10:14:48.489 [LogStash::Runner] runner - queue.checkpoint.acks: 1024
[DEBUG] 2022-03-28 10:14:48.489 [LogStash::Runner] runner - queue.checkpoint.writes: 1024
[DEBUG] 2022-03-28 10:14:48.489 [LogStash::Runner] runner - queue.checkpoint.interval: 1000
[DEBUG] 2022-03-28 10:14:48.489 [LogStash::Runner] runner - queue.checkpoint.retry: false
[DEBUG] 2022-03-28 10:14:48.489 [LogStash::Runner] runner - dead_letter_queue.enable: false
[DEBUG] 2022-03-28 10:14:48.489 [LogStash::Runner] runner - dead_letter_queue.max_bytes: 1073741824
[DEBUG] 2022-03-28 10:14:48.489 [LogStash::Runner] runner - dead_letter_queue.flush_interval: 5000
[DEBUG] 2022-03-28 10:14:48.489 [LogStash::Runner] runner - slowlog.threshold.warn: #<Java::OrgLogstashUtil::TimeValue:0x28a972dd>
[DEBUG] 2022-03-28 10:14:48.490 [LogStash::Runner] runner - slowlog.threshold.info: #<Java::OrgLogstashUtil::TimeValue:0x52651c27>
[DEBUG] 2022-03-28 10:14:48.490 [LogStash::Runner] runner - slowlog.threshold.debug: #<Java::OrgLogstashUtil::TimeValue:0x41600f6b>
[DEBUG] 2022-03-28 10:14:48.490 [LogStash::Runner] runner - slowlog.threshold.trace: #<Java::OrgLogstashUtil::TimeValue:0x6ed1925>
[DEBUG] 2022-03-28 10:14:48.490 [LogStash::Runner] runner - keystore.classname: "org.logstash.secret.store.backend.JavaKeyStore"
[DEBUG] 2022-03-28 10:14:48.490 [LogStash::Runner] runner - keystore.file: "/usr/share/logstash/config/logstash.keystore"
[DEBUG] 2022-03-28 10:14:48.490 [LogStash::Runner] runner - path.queue: "/usr/share/logstash/data/queue"
[DEBUG] 2022-03-28 10:14:48.490 [LogStash::Runner] runner - path.dead_letter_queue: "/usr/share/logstash/data/dead_letter_queue"
[DEBUG] 2022-03-28 10:14:48.490 [LogStash::Runner] runner - path.settings: "/usr/share/logstash/config"
[DEBUG] 2022-03-28 10:14:48.490 [LogStash::Runner] runner - path.logs: "/usr/share/logstash/logs"
[DEBUG] 2022-03-28 10:14:48.490 [LogStash::Runner] runner - xpack.monitoring.enabled: false
[DEBUG] 2022-03-28 10:14:48.490 [LogStash::Runner] runner - xpack.monitoring.elasticsearch.hosts: ["http://localhost:9200"]
[DEBUG] 2022-03-28 10:14:48.491 [LogStash::Runner] runner - xpack.monitoring.collection.interval: #<Java::OrgLogstashUtil::TimeValue:0x61f844c0>
[DEBUG] 2022-03-28 10:14:48.491 [LogStash::Runner] runner - xpack.monitoring.collection.timeout_interval: #<Java::OrgLogstashUtil::TimeValue:0x6338b2f0>
[DEBUG] 2022-03-28 10:14:48.491 [LogStash::Runner] runner - xpack.monitoring.elasticsearch.username: "logstash_system"
[DEBUG] 2022-03-28 10:14:48.491 [LogStash::Runner] runner - xpack.monitoring.elasticsearch.ssl.verification_mode: "certificate"
[DEBUG] 2022-03-28 10:14:48.491 [LogStash::Runner] runner - xpack.monitoring.elasticsearch.sniffing: false
[DEBUG] 2022-03-28 10:14:48.491 [LogStash::Runner] runner - xpack.monitoring.collection.pipeline.details.enabled: true
[DEBUG] 2022-03-28 10:14:48.491 [LogStash::Runner] runner - xpack.monitoring.collection.config.enabled: true
[DEBUG] 2022-03-28 10:14:48.491 [LogStash::Runner] runner - monitoring.enabled: false
[DEBUG] 2022-03-28 10:14:48.491 [LogStash::Runner] runner - monitoring.elasticsearch.hosts: ["http://localhost:9200"]
[DEBUG] 2022-03-28 10:14:48.491 [LogStash::Runner] runner - monitoring.collection.interval: #<Java::OrgLogstashUtil::TimeValue:0x22e628bb>
[DEBUG] 2022-03-28 10:14:48.492 [LogStash::Runner] runner - monitoring.collection.timeout_interval: #<Java::OrgLogstashUtil::TimeValue:0x43060188>
[DEBUG] 2022-03-28 10:14:48.492 [LogStash::Runner] runner - monitoring.elasticsearch.username: "logstash_system"
[DEBUG] 2022-03-28 10:14:48.492 [LogStash::Runner] runner - monitoring.elasticsearch.ssl.verification_mode: "certificate"
[DEBUG] 2022-03-28 10:14:48.492 [LogStash::Runner] runner - monitoring.elasticsearch.sniffing: false
[DEBUG] 2022-03-28 10:14:48.492 [LogStash::Runner] runner - monitoring.collection.pipeline.details.enabled: true
[DEBUG] 2022-03-28 10:14:48.492 [LogStash::Runner] runner - monitoring.collection.config.enabled: true
[DEBUG] 2022-03-28 10:14:48.492 [LogStash::Runner] runner - node.uuid: ""
[DEBUG] 2022-03-28 10:14:48.492 [LogStash::Runner] runner - xpack.management.enabled: false
[DEBUG] 2022-03-28 10:14:48.492 [LogStash::Runner] runner - xpack.management.logstash.poll_interval: #<Java::OrgLogstashUtil::TimeValue:0x58a5a78f>
[DEBUG] 2022-03-28 10:14:48.492 [LogStash::Runner] runner - xpack.management.pipeline.id: ["main"]
[DEBUG] 2022-03-28 10:14:48.492 [LogStash::Runner] runner - xpack.management.elasticsearch.username: "logstash_system"
[DEBUG] 2022-03-28 10:14:48.493 [LogStash::Runner] runner - xpack.management.elasticsearch.hosts: ["https://localhost:9200"]
[DEBUG] 2022-03-28 10:14:48.493 [LogStash::Runner] runner - xpack.management.elasticsearch.ssl.verification_mode: "certificate"
[DEBUG] 2022-03-28 10:14:48.493 [LogStash::Runner] runner - xpack.management.elasticsearch.sniffing: false
[DEBUG] 2022-03-28 10:14:48.493 [LogStash::Runner] runner - --------------- Logstash Settings -------------------
[WARN ] 2022-03-28 10:14:49.179 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[DEBUG] 2022-03-28 10:14:49.242 [LogStash::Runner] configpathloader - Skipping the following files while reading config since they don't match the specified glob pattern {:files=>["/etc/DIR_COLORS", "/etc/DIR_COLORS.256color", "/etc/DIR_COLORS.lightbgcolor", "/etc/GREP_COLORS", "/etc/GeoIP.conf", "/etc/NetworkManager", "/etc/X11", "/etc/adjtime", "/etc/aide.conf", "/etc/aliases", "/etc/aliases.db", "/etc/aliases.rpmnew", "/etc/alternatives", "/etc/anacrontab", "/etc/asound.conf", "/etc/at.allow", "/etc/audisp", "/etc/audit", "/etc/auto.home", "/etc/auto.home-TDS", "/etc/auto.master", "/etc/auto.master.d", "/etc/auto.misc", "/etc/auto.net", "/etc/auto.smb", "/etc/autofs.conf", "/etc/autofs_ldap_auth.conf", "/etc/bash_completion.d", "/etc/bashrc", "/etc/binfmt.d", "/etc/centos-release", "/etc/centos-release-upstream", "/etc/chkconfig.d", "/etc/cifs-utils", "/etc/cron.allow", "/etc/cron.d", "/etc/cron.daily", "/etc/cron.deny", "/etc/cron.hourly", "/etc/cron.monthly", "/etc/cron.weekly", "/etc/crontab", "/etc/crypttab", "/etc/csh.cshrc", "/etc/csh.login", "/etc/dbus-1", "/etc/default", "/etc/depmod.d", "/etc/dhcp", "/etc/dnf", "/etc/dracut.conf", "/etc/dracut.conf.d", "/etc/duo_log_sync", "/etc/e2fsck.conf", "/etc/egl", "/etc/environment", "/etc/ethertypes", "/etc/exports", "/etc/exports.d", "/etc/favicon.png", "/etc/filebeat", "/etc/filesystems", "/etc/firewalld", "/etc/fonts", "/etc/freshclam.conf.rpmsave", "/etc/fstab", "/etc/fuse.conf", "/etc/gcrypt", "/etc/glvnd", "/etc/gnupg", "/etc/gofer", "/etc/groff", "/etc/group", "/etc/group-", "/etc/grub.d", "/etc/grub2.cfg", "/etc/gshadow", "/etc/gshadow-", "/etc/gss", "/etc/gssproxy", "/etc/host.conf", "/etc/hostname", "/etc/hosts", "/etc/hosts.allow", "/etc/hosts.deny", "/etc/idmapd.conf", "/etc/init.d", "/etc/inittab", "/etc/inputrc", "/etc/iproute2", "/etc/issue", "/etc/issue.net", "/etc/java", "/etc/jvm", "/etc/jvm-commmon", "/etc/kdump.conf", "/etc/kernel", "/etc/krb5.conf", "/etc/krb5.conf.d", "/etc/krb5.keytab", "/etc/ld.so.cache", "/etc/ld.so.conf", "/etc/ld.so.conf.d", "/etc/libaudit.conf", "/etc/libnl", "/etc/libuser.conf", "/etc/locale.conf", "/etc/localtime", "/etc/login.defs", "/etc/login.defs.rpmnew", "/etc/logrotate.conf", "/etc/logrotate.d", "/etc/logstash", "/etc/lvm", "/etc/machine-id", "/etc/magic", "/etc/mail.rc", "/etc/makedumpfile.conf.sample", "/etc/man_db.conf", "/etc/maven", "/etc/mke2fs.conf", "/etc/modprobe.d", "/etc/modules-load.d", "/etc/motd", "/etc/mtab", "/etc/my.cnf", "/etc/my.cnf.d", "/etc/netconfig", "/etc/networks", "/etc/nfs.conf", "/etc/nfsmount.conf", "/etc/nsswitch.conf", "/etc/nsswitch.conf.bak", "/etc/nsswitch.conf.rpmnew", "/etc/ntp", "/etc/ntp.conf", "/etc/oddjob", "/etc/oddjobd.conf", "/etc/oddjobd.conf.d", "/etc/openldap", "/etc/opt", "/etc/os-release", "/etc/pam.d", "/etc/passwd", "/etc/passwd-", "/etc/pkcs11", "/etc/pki", "/etc/plymouth", "/etc/pm", "/etc/polkit-1", "/etc/popt.d", "/etc/postfix", "/etc/ppp", "/etc/prelink.conf.d", "/etc/printcap", "/etc/profile", "/etc/profile.d", "/etc/protocols", "/etc/python", "/etc/rc.d", "/etc/rc.local", "/etc/rc0.d", "/etc/rc1.d", "/etc/rc2.d", "/etc/rc3.d", "/etc/rc4.d", "/etc/rc5.d", "/etc/rc6.d", "/etc/realmd.conf", "/etc/redhat-release", "/etc/request-key.conf", "/etc/request-key.d", "/etc/resolv.conf", "/etc/resolv.conf.save", "/etc/rhsm", "/etc/rpc", "/etc/rpm", "/etc/rsyncd.conf", "/etc/rsyslog.conf", "/etc/rsyslog.d", "/etc/rwtab", "/etc/rwtab.d", "/etc/samba", "/etc/sasl2", "/etc/screenrc", "/etc/securetty", "/etc/security", "/etc/selinux", "/etc/services", "/etc/sestatus.conf", "/etc/shadow", "/etc/shadow-", "/etc/shells", "/etc/skel", "/etc/snmp", "/etc/ssh", "/etc/ssl", "/etc/sssd", "/etc/statetab", "/etc/statetab.d", "/etc/subgid", "/etc/subgid-", "/etc/subuid", "/etc/subuid-", "/etc/sudo-ldap.conf", "/etc/sudo.conf", "/etc/sudoers", "/etc/sudoers.d", "/etc/sysconfig", "/etc/sysctl.conf", "/etc/sysctl.d", "/etc/system-release", "/etc/system-release-cpe", "/etc/systemd", "/etc/tcsd.conf", "/etc/tds-release", "/etc/terminfo", "/etc/tmpfiles.d", "/etc/trusted-key.key", "/etc/tuned", "/etc/udev", "/etc/updatedb.conf", "/etc/vconsole.conf", "/etc/vimrc", "/etc/virc", "/etc/vmware-caf", "/etc/vmware-tools", "/etc/wgetrc", "/etc/wpa_supplicant", "/etc/xdg", "/etc/xinetd.d", "/etc/yum", "/etc/yum.conf", "/etc/yum.repos.d"]}
[DEBUG] 2022-03-28 10:14:49.248 [LogStash::Runner] configpathloader - Reading config file {:config_file=>"/etc/logstash/jvm.options"}
[DEBUG] 2022-03-28 10:14:49.254 [LogStash::Runner] configpathloader - Reading config file {:config_file=>"/etc/logstash/log4j2.properties"}
[DEBUG] 2022-03-28 10:14:49.278 [LogStash::Runner] configpathloader - Reading config file {:config_file=>"/etc/logstash/logstash.yml"}
[DEBUG] 2022-03-28 10:14:49.279 [LogStash::Runner] configpathloader - Reading config file {:config_file=>"/etc/logstash/pipelines.yml"}
[DEBUG] 2022-03-28 10:14:49.292 [LogStash::Runner] configpathloader - Reading config file {:config_file=>"/etc/logstash/startup.options"}
[DEBUG] 2022-03-28 10:14:49.324 [LogStash::Runner] SecretStoreFactory - Attempting to exists or secret store with implementation: org.logstash.secret.store.backend.JavaKeyStore
[FATAL] 2022-03-28 10:14:49.499 [LogStash::Runner] runner - The given configuration is invalid. Reason: Expected one of [ \t\r\n], "#", "input", "filter", "output" at line 6, column 1 (byte 132) after
[FATAL] 2022-03-28 10:14:49.502 [LogStash::Runner] Logstash - Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
	at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby.jar:?]
	at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby.jar:?]
	at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]

Badger · March 28, 2022, 3:10pm

That is your problem. You should point -f at a file, or at a directory that only contains logstash pipeline configurations. /etc/logstash contains several other files...

[DEBUG] 2022-03-28 10:14:49.248 [LogStash::Runner] configpathloader - Reading config file {:config_file=>"/etc/logstash/jvm.options"}
[DEBUG] 2022-03-28 10:14:49.254 [LogStash::Runner] configpathloader - Reading config file {:config_file=>"/etc/logstash/log4j2.properties"}
[DEBUG] 2022-03-28 10:14:49.278 [LogStash::Runner] configpathloader - Reading config file {:config_file=>"/etc/logstash/logstash.yml"}
[DEBUG] 2022-03-28 10:14:49.279 [LogStash::Runner] configpathloader - Reading config file {:config_file=>"/etc/logstash/pipelines.yml"}
[DEBUG] 2022-03-28 10:14:49.292 [LogStash::Runner] configpathloader - Reading config file {:config_file=>"/etc/logstash/startup.options"}

marone · March 29, 2022, 4:36pm

Hey all,

I have the same error, upgraded from logstash 7.17 to 8.1, logstash is running as a service, I tested first with a simple logstash configuration file:

input { stdin { } }
output {
  elasticsearch { hosts => ["myhost2:9200"]
  user => "logstash_internal"
  password => "my_pw"
  ssl => true
  cacert => '/etc/logstash/config/certs/elasticsearch-ca.pem'
  ssl_certificate_verification => false
}
  stdout { codec => rubydebug }
}

we was using a self-signed certification since 7.13, the command line to test logstash: /usr/share/logstash/bin/logstash -f conf.d/test.conf --log.level=debug

Then I have the following error:

.....
[DEBUG] 2022-03-29 16:34:53.136 [LogStash::Runner] runner - xpack.management.elasticsearch.ssl.verification_mode: "certificate"
[DEBUG] 2022-03-29 16:34:53.136 [LogStash::Runner] runner - xpack.management.elasticsearch.sniffing: false
[DEBUG] 2022-03-29 16:34:53.136 [LogStash::Runner] runner - --------------- Logstash Settings -------------------
[WARN ] 2022-03-29 16:34:53.215 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[FATAL] 2022-03-29 16:34:53.239 [LogStash::Runner] runner - An unexpected error occurred! {:error=>java.nio.file.AccessDeniedException: /usr/share/logstash/data/.lock, :backtrace=>["sun.nio.fs.UnixException.translateToIOException(sun/nio/fs/UnixException.java:90)", "sun.nio.fs.UnixException.rethrowAsIOException(sun/nio/fs/UnixException.java:111)", "sun.nio.fs.UnixException.rethrowAsIOException(sun/nio/fs/UnixException.java:116)", "sun.nio.fs.UnixFileSystemProvider.newFileChannel(sun/nio/fs/UnixFileSystemProvider.java:182)", "java.nio.channels.FileChannel.open(java/nio/channels/FileChannel.java:292)", "java.nio.channels.FileChannel.open(java/nio/channels/FileChannel.java:345)", "org.logstash.FileLockFactory.obtainLock(org/logstash/FileLockFactory.java:89)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:62)", "jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:566)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:471)", "org.jruby.javasupport.JavaMethod.invokeStaticDirect(org/jruby/javasupport/JavaMethod.java:373)", "RUBY.execute(/usr/share/logstash/logstash-core/lib/logstash/runner.rb:396)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.clamp_minus_1_dot_0_dot_1.lib.clamp.command.run(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-1.0.1/lib/clamp/command.rb:68)", "usr.share.logstash.logstash_minus_core.lib.logstash.runner.run(/usr/share/logstash/logstash-core/lib/logstash/runner.rb:282)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.clamp_minus_1_dot_0_dot_1.lib.clamp.command.run(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-1.0.1/lib/clamp/command.rb:133)", "usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:93)", "java.lang.invoke.MethodHandle.invokeWithArguments(java/lang/invoke/MethodHandle.java:710)", "org.jruby.Ruby.runScript(org/jruby/Ruby.java:1205)", "org.jruby.Ruby.runNormally(org/jruby/Ruby.java:1128)", "org.jruby.Ruby.runNormally(org/jruby/Ruby.java:1146)", "org.jruby.Ruby.runFromMain(org/jruby/Ruby.java:958)", "org.logstash.Logstash.run(org/logstash/Logstash.java:163)", "org.logstash.Logstash.main(org/logstash/Logstash.java:73)"]}
[FATAL] 2022-03-29 16:34:53.258 [LogStash::Runner] Logstash - Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby.jar:?]
        at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]

Any idea please?

leandrojmp · March 29, 2022, 4:41pm

Did you run logstash as a root while testing? This is a permissions issue, it normally happens when logstash is run by the root user and started as a service, which uses the logstash user.

marone · March 29, 2022, 4:44pm

yes I changed the ownership to logstash user, and just realized the access denied before, gave to .lock write permission and worked, but then logstash terminate, saying: No config files found in path {:path=>"/usr/share/logstash/conf.d/test.conf" but there is a pipeline configuration file ^^

leandrojmp · March 29, 2022, 4:50pm

I would suggest that you open another topic as this one is already marked as solved.

Also, share your logstash.yml and pipelines.yml files in the new topic.

system · April 26, 2022, 4:50pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Logstash error during configuration Logstash	5	1038	March 1, 2022
Logstash Logstash	2	399	May 12, 2021
Help please: logstash start error Logstash	3	355	April 16, 2021
Logstash output question Logstash	4	194	August 4, 2022
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release Logstash	3	13060	June 13, 2022

Default install of logstash 8.1 results in error

Related Topics