Filebeat won't start

its-ogawa · March 2, 2021, 1:39am

I'm trying to link filebeat with logstash, but filebeat suddenly stopped working.
When I try systemctl start filebeat, it does not start.

If I run systemctl status filebeat, I get the following message.

● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
   Loaded: loaded (/usr/lib/systemd/system/filebeat.service; disabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since 月 2021-03-01 19:58:49 JST; 13h ago
     Docs: https://www.elastic.co/products/beats/filebeat
  Process: 7200 ExecStart=/usr/share/filebeat/bin/filebeat --environment systemd $BEAT_LOG_OPTS $BEAT_CONFIG_OPTS $BEAT_PATH_OPTS (code=exited, status=1/FAILURE)
 Main PID: 7200 (code=exited, status=1/FAILURE)

 3月 01 19:58:49 ITS-ELS-01 filebeat[7200]: runtime.goexit
 3月 01 19:58:49 ITS-ELS-01 filebeat[7200]: /usr/local/go/src/runtime/asm_amd64.s:1373
 3月 01 19:58:49 ITS-ELS-01 systemd[1]: Unit filebeat.service entered failed state.
 3月 01 19:58:49 ITS-ELS-01 systemd[1]: filebeat.service failed.
 3月 01 19:58:49 ITS-ELS-01 systemd[1]: filebeat.service holdoff time over, scheduling restart.
 3月 01 19:58:49 ITS-ELS-01 systemd[1]: Stopped Filebeat sends log files to Logstash or directly to Elasticsearch..
 3月 01 19:58:49 ITS-ELS-01 systemd[1]: start request repeated too quickly for filebeat.service
 3月 01 19:58:49 ITS-ELS-01 systemd[1]: Failed to start Filebeat sends log files to Logstash or directly to Elasticsearch..
 3月 01 19:58:49 ITS-ELS-01 systemd[1]: Unit filebeat.service entered failed state.
 3月 01 19:58:49 ITS-ELS-01 systemd[1]: filebeat.service failed.

"start request repeated too quickly for filebeat.service".
I have restarted the service overnight, but the situation remains the same.

What do I need to do?

warkolm · March 2, 2021, 2:06am

What do the actual Filebeat logs show you?

its-ogawa · March 2, 2021, 2:08am

Thanks for the reply.

The filebeat logs do not contain any significant messages.

# cat /var/log/filebeat/filebeat
2021-03-01T17:53:59.021+0900    INFO    instance/beat.go:660    Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2021-03-01T17:53:59.022+0900    INFO    instance/beat.go:668    Beat ID: b3925fa0-2599-40f2-a95f-bb0557f6e600

warkolm · March 2, 2021, 2:09am

Can you try starting Filebeat directly as a command?

its-ogawa · March 2, 2021, 2:13am

Is it what you expect?

# /usr/share/logstash/bin/logstash
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2021-03-02 11:10:47.142 [main] runner - Starting Logstash {"logstash.version"=>"7.11.0", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.8+10 on 11.0.8+10 +indy +jit [linux-x86_64]"}
ERROR: Failed to read pipelines yaml file. Location: /usr/share/logstash/config/pipelines.yml
usage:
  bin/logstash -f CONFIG_PATH [-t] [-r] [] [-w COUNT] [-l LOG]
  bin/logstash --modules MODULE_NAME [-M "MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAME.VARIABLE_NAME=VALUE"] [-t] [-w COUNT] [-l LOG]
  bin/logstash -e CONFIG_STR [-t] [--log.level fatal|error|warn|info|debug|trace] [-w COUNT] [-l LOG]
  bin/logstash -i SHELL [--log.level fatal|error|warn|info|debug|trace]
  bin/logstash -V [--log.level fatal|error|warn|info|debug|trace]
  bin/logstash --help
[FATAL] 2021-03-02 11:10:47.490 [LogStash::Runner] Logstash - Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.13.0.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.13.0.jar:?]
        at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:89) ~[?:?]

warkolm · March 2, 2021, 2:22am

Your post is about Filebeat but this is Logstash?

its-ogawa · March 2, 2021, 2:26am

I'm sorry. It's my mistake.
Let me retract my earlier post.

Is it what you expect?

#filebeat
Exiting: Failed to start crawler: creating module reloader failed: 1 error: invalid config: yaml: line 13: could not find expected ':'
loading configs
github.com/elastic/beats/v7/libbeat/cfgfile.(*Reloader).Check
        /go/src/github.com/elastic/beats/libbeat/cfgfile/reload.go:143
github.com/elastic/beats/v7/filebeat/beater.(*crawler).Start
        /go/src/github.com/elastic/beats/filebeat/beater/crawler.go:91
github.com/elastic/beats/v7/filebeat/beater.(*Filebeat).Run
        /go/src/github.com/elastic/beats/filebeat/beater/filebeat.go:438
github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch
        /go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:474
github.com/elastic/beats/v7/libbeat/cmd/instance.Run.func1
        /go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:193
github.com/elastic/beats/v7/libbeat/cmd/instance.Run
        /go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:194
github.com/elastic/beats/v7/libbeat/cmd.genRunCmd.func1
        /go/src/github.com/elastic/beats/libbeat/cmd/run.go:36
github.com/spf13/cobra.(*Command).execute
        /go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830
github.com/spf13/cobra.(*Command).ExecuteC
        /go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
        /go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
main.main
        /go/src/github.com/elastic/beats/x-pack/filebeat/main.go:22
runtime.main
        /usr/local/go/src/runtime/proc.go:203
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1373

warkolm · March 2, 2021, 2:26am

That seems relevant.
What does your config look like.

its-ogawa · March 2, 2021, 2:32am

Checked filebeat.yml.
I'm sorry, but I don't see where it is wrong.
I will present all the contents.

# cat /etc/filebeat/filebeat.yml
filebeat.inputs:
#- type: log
#  enabled: true
#  paths:
#    - /usr/local/elastic/datasets/logstash-tutorial.log
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
#- type: log#nginx-error
#  enabled: true
#  paths:
#    - /var/log/nginx/error.log
#- type: log#messages
#  enabled: true
#  paths:
#    - /var/log/messages


output.logstash:
  hosts: ["http://***.***.***.***:5044"]

filebeat.config.modules:
  enabled: true
  path: ${path.config}/modules.d/*.yml

The relevant line is commented out, but does line 13 count only valid lines?

its-ogawa · March 2, 2021, 9:14am

Did I write the wrong yml again?

I'll also write nginx.yml just in case.
However, I can't find any error in line 13 here either.

# cat /etc/filebeat/modules.d/nginx.yml
# Module: nginx
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.11/filebeat-module-nginx.html

- module: nginx
  # Access logs
  access:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths:["/var/log/nginx/access.log"]

  # Error logs
  error:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths:["/var/log/nginx/error.log"]

  # Ingress-nginx controller logs. This is disabled by default. It could be used in Kubernetes environments to parse ingress-nginx logs
  ingress_controller:
    enabled: false

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

its-ogawa · March 2, 2021, 11:22am

Oh. I finally understood.
It seems that the nginx.yml was written incorrectly.
The reason is that there is no "space" after ":".
That's all there was to it.

However, if you look at the runtime error of filebeat, it doesn't seem to represent the error accurately.
The error was as follows:
Exiting: Failed to start crawler: creating module reloader failed: 1 error: invalid config: yaml: line 13: could not find expected ':' loading configs

Don't be fooled by "line 13: could not find expected ':'".
Look carefully at "/etc/filebeat/modules.d/nginx.yml".

We know that these problems are difficult to detect correctly. We know that these problems are difficult to detect correctly, but if you could review the error detection and messages, it would make the application even better.

its-ogawa · March 2, 2021, 11:31am

For your reference, here are the events we have experienced so far, how to check the cause, and where we have modified.
Just as a reminder.

Incident:

Filebeat won’t start.

# systemctl status filebeat
● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
   Loaded: loaded (/usr/lib/systemd/system/filebeat.service; enabled; vendor preset: disabled)
   ... snip ...

It says here that there are the following errors.

Stopped Filebeat sends log files to Logstash or directly to Elasticsearch..
start request repeated too quickly for filebeat.service
Failed to start Filebeat sends log files to Logstash or directly to Elasticsearch..
Unit filebeat.service entered failed state.

Confirmation:

View the filebeat runtime log.

# filebeat -e -v
... snip ...
2021-03-02T19:57:39.510+0900    ERROR   cfgfile/reload.go:273   Error loading config from file '/etc/filebeat/modules.d/nginx.yml', error invalid config: yaml: line 13: could not find expected ':'
2021-03-02T19:57:39.511+0900    INFO    beater/crawler.go:148   Stopping Crawler
2021-03-02T19:57:39.511+0900    INFO    beater/crawler.go:158   Stopping 1 inputs
2021-03-02T19:57:39.511+0900    INFO    [crawler]       beater/crawler.go:163   Stopping input: 11653597076401989289
2021-03-02T19:57:39.511+0900    ERROR   log/input.go:542        Harvester could not be started on existing file: /var/log/nginx/access.log, Err: registry already stopped
2021-03-02T19:57:39.511+0900    INFO    log/input.go:488        Scan aborted because input stopped.
2021-03-02T19:57:39.511+0900    INFO    input/input.go:136      input ticker stopped
2021-03-02T19:57:39.511+0900    INFO    beater/crawler.go:178   Crawler stopped
... snip ...
2021-03-02T19:57:39.515+0900    ERROR   instance/beat.go:971    Exiting: Failed to start crawler: creating module reloader failed: 1 error: invalid config: yaml: line 13: could not find expected ':'
loading configs
... snip ...

It says here that there are the following errors.
Error loading config from file '/etc/filebeat/modules.d/nginx.yml', error invalid config: yaml: line 13: could not find expected ':'

Normally, this error is not enough to tell you what is going on, but it does tell you that there may be an error in this file.

Fixes:

Put a "space" after ":".

# cat /etc/filebeat/modules.d/nginx.yml
# Module: nginx
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.11/filebeat-module-nginx.html

- module: nginx
  # Access logs
  access:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/nginx/access.log"]

  # Error logs
  error:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/nginx/error.log"]

  # Ingress-nginx controller logs. This is disabled by default. It could be used in Kubernetes environments to parse ingress-nginx logs
  ingress_controller:
    enabled: false

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

Results:

You can start filebeat.

# systemctl start filebeat
# systemctl status filebeat
● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
   Loaded: loaded (/usr/lib/systemd/system/filebeat.service; enabled; vendor preset: disabled)
   ... snip ...

That's it.

system · March 30, 2021, 1:31pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.