# /usr/share/logstash/bin/logstash
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2021-03-02 11:10:47.142 [main] runner - Starting Logstash {"logstash.version"=>"7.11.0", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.8+10 on 11.0.8+10 +indy +jit [linux-x86_64]"}
ERROR: Failed to read pipelines yaml file. Location: /usr/share/logstash/config/pipelines.yml
usage:
bin/logstash -f CONFIG_PATH [-t] [-r] [] [-w COUNT] [-l LOG]
bin/logstash --modules MODULE_NAME [-M "MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAME.VARIABLE_NAME=VALUE"] [-t] [-w COUNT] [-l LOG]
bin/logstash -e CONFIG_STR [-t] [--log.level fatal|error|warn|info|debug|trace] [-w COUNT] [-l LOG]
bin/logstash -i SHELL [--log.level fatal|error|warn|info|debug|trace]
bin/logstash -V [--log.level fatal|error|warn|info|debug|trace]
bin/logstash --help
[FATAL] 2021-03-02 11:10:47.490 [LogStash::Runner] Logstash - Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.13.0.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.13.0.jar:?]
at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:89) ~[?:?]
I'll also write nginx.yml just in case.
However, I can't find any error in line 13 here either.
# cat /etc/filebeat/modules.d/nginx.yml
# Module: nginx
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.11/filebeat-module-nginx.html
- module: nginx
# Access logs
access:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
var.paths:["/var/log/nginx/access.log"]
# Error logs
error:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
var.paths:["/var/log/nginx/error.log"]
# Ingress-nginx controller logs. This is disabled by default. It could be used in Kubernetes environments to parse ingress-nginx logs
ingress_controller:
enabled: false
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:
Oh. I finally understood.
It seems that the nginx.yml was written incorrectly. The reason is that there is no "space" after ":".
That's all there was to it.
However, if you look at the runtime error of filebeat, it doesn't seem to represent the error accurately.
The error was as follows: Exiting: Failed to start crawler: creating module reloader failed: 1 error: invalid config: yaml: line 13: could not find expected ':' loading configs
Don't be fooled by "line 13: could not find expected ':'". Look carefully at "/etc/filebeat/modules.d/nginx.yml".
We know that these problems are difficult to detect correctly. We know that these problems are difficult to detect correctly, but if you could review the error detection and messages, it would make the application even better.
For your reference, here are the events we have experienced so far, how to check the cause, and where we have modified.
Just as a reminder.
Incident:
Filebeat won’t start.
# systemctl status filebeat
● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
Loaded: loaded (/usr/lib/systemd/system/filebeat.service; enabled; vendor preset: disabled)
... snip ...
It says here that there are the following errors.
Stopped Filebeat sends log files to Logstash or directly to Elasticsearch..
start request repeated too quickly for filebeat.service
Failed to start Filebeat sends log files to Logstash or directly to Elasticsearch..
Unit filebeat.service entered failed state.
Confirmation:
View the filebeat runtime log.
# filebeat -e -v
... snip ...
2021-03-02T19:57:39.510+0900 ERROR cfgfile/reload.go:273 Error loading config from file '/etc/filebeat/modules.d/nginx.yml', error invalid config: yaml: line 13: could not find expected ':'
2021-03-02T19:57:39.511+0900 INFO beater/crawler.go:148 Stopping Crawler
2021-03-02T19:57:39.511+0900 INFO beater/crawler.go:158 Stopping 1 inputs
2021-03-02T19:57:39.511+0900 INFO [crawler] beater/crawler.go:163 Stopping input: 11653597076401989289
2021-03-02T19:57:39.511+0900 ERROR log/input.go:542 Harvester could not be started on existing file: /var/log/nginx/access.log, Err: registry already stopped
2021-03-02T19:57:39.511+0900 INFO log/input.go:488 Scan aborted because input stopped.
2021-03-02T19:57:39.511+0900 INFO input/input.go:136 input ticker stopped
2021-03-02T19:57:39.511+0900 INFO beater/crawler.go:178 Crawler stopped
... snip ...
2021-03-02T19:57:39.515+0900 ERROR instance/beat.go:971 Exiting: Failed to start crawler: creating module reloader failed: 1 error: invalid config: yaml: line 13: could not find expected ':'
loading configs
... snip ...
It says here that there are the following errors. Error loading config from file '/etc/filebeat/modules.d/nginx.yml', error invalid config: yaml: line 13: could not find expected ':'
Normally, this error is not enough to tell you what is going on, but it does tell you that there may be an error in this file.
Fixes:
Put a "space" after ":".
# cat /etc/filebeat/modules.d/nginx.yml
# Module: nginx
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.11/filebeat-module-nginx.html
- module: nginx
# Access logs
access:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
var.paths: ["/var/log/nginx/access.log"]
# Error logs
error:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
var.paths: ["/var/log/nginx/error.log"]
# Ingress-nginx controller logs. This is disabled by default. It could be used in Kubernetes environments to parse ingress-nginx logs
ingress_controller:
enabled: false
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:
Results:
You can start filebeat.
# systemctl start filebeat
# systemctl status filebeat
● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
Loaded: loaded (/usr/lib/systemd/system/filebeat.service; enabled; vendor preset: disabled)
... snip ...
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.