Filebeat 6.5.1 unable to send syslog

princeOfMacedon · November 22, 2018, 8:17am

Hi all!

I'm using filebeat 6.5.1 and I'm encountering this error:

filebeat[29062]: Exiting: Can only start an input when all related states are finished: {Id:2537-66305 Finished:false Fileinfo:0xc420213860 Source:/var/log/auth.log Offset:618294 Timestamp:2018-11-22 07:10:05.795725847 +0000 UTC m=+0.072986119 TTL:-1ns Type:log Meta:map FileS

I follow the instruction here Filebeat: Exiting: Can only start an input when all related states are finished and delete /var/lib/filebeat/registry.

Before deleting, the registry content is:

[{"source":"/var/log/cloud-init-output.log","offset":4850,"timestamp":"2018-11-13T05:21:23.36633439Z","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":2974,"device":66305}},{"source":"/var/log/cloud-init.log","offset":147844,"timestamp":"2018-11-13T05:21:23.400354561Z","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":2971,"device":66305}},{"source":"/var/log/dpkg.log","offset":152397,"timestamp":"2018-11-21T02:59:00.553800623Z","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":58290,"device":66305}},{"source":"/var/log/kern.log","offset":199276,"timestamp":"2018-11-22T07:04:14.668430496Z","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":17629,"device":66305}},{"source":"/var/log/alternatives.log","offset":528,"timestamp":"2018-11-13T05:21:48.366825045Z","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":2328,"device":66305}},{"source":"/var/log/auth.log","offset":694771,"timestamp":"2018-11-22T07:22:00.6795606Z","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":17713,"device":66305}},{"source":"/var/log/syslog","offset":22833471,"timestamp":"2018-11-22T07:25:39.441186992Z","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":2366,"device":66305}},{"source":"/var/log/syslog.1","offset":525329504,"timestamp":"2018-11-22T06:30:17.939994178Z","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":2358,"device":66305}},{"source":"/var/log/syslog.2","offset":196173115,"timestamp":"2018-11-22T06:54:22.880009798Z","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":2309,"device":66305}}]

After that, I restart filebeat, it show no error, but no syslog is pushed to elastic while nginx log still works as expected.

The registry content is now:

[{"source":"/var/log/dpkg.log","offset":152397,"timestamp":"2018-11-22T07:52:15.145040968Z","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":58290,"device":66305}},{"source":"/var/log/kern.log","offset":199276,"timestamp":"2018-11-22T07:52:18.005431008Z","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":17629,"device":66305}},{"source":"/var/log/alternatives.log","offset":528,"timestamp":"2018-11-22T07:52:13.698411707Z","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":2328,"device":66305}},{"source":"/var/log/auth.log","offset":703210,"timestamp":"2018-11-22T07:52:30.223743183Z","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":17713,"device":66305}},{"source":"/var/log/cloud-init-output.log","offset":4850,"timestamp":"2018-11-22T07:52:13.763932619Z","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":2974,"device":66305}},{"source":"/var/log/cloud-init.log","offset":147844,"timestamp":"2018-11-22T07:52:15.912014197Z","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":2971,"device":66305}},{"source":"/var/log/nginx/access.log","offset":45367,"timestamp":"2018-11-22T07:55:02.519360467Z","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":515336,"device":66305}},{"source":"/var/log/nginx/access.log.1","offset":781856,"timestamp":"2018-11-22T07:52:21.674422476Z","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":516014,"device":66305}},{"source":"/var/log/nginx/error.log","offset":0,"timestamp":"2018-11-22T07:52:13.686014045Z","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":515668,"device":66305}},{"source":"/var/log/nginx/error.log.1","offset":4648,"timestamp":"2018-11-22T07:52:15.205459069Z","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":516326,"device":66305}}]

Thanks in advanced!

princeOfMacedon · November 22, 2018, 8:21am

Here is my /etc/filebeat/module.d/system.yml

- module: system
  # Syslog
  syslog:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

    # Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
    var.convert_timezone: true

  # Authorization logs
  auth:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

    # Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
    var.convert_timezone: true

Here is a part of my /etc/filebeat/filebeat.yml:

- type: log

  # Change to true to enable this input configuration.

enabled: true

  # Paths that should be crawled and fetched. Glob based paths.

paths:

- /var/log/*.log

#- c:\programdata\elasticsearch\logs\*

  # Exclude lines. A list of regular expressions to match. It drops the lines that are

  # matching any regular expression from the list.

  #exclude_lines: ['^DBG']

  # Include lines. A list of regular expressions to match. It exports the lines that are

  # matching any regular expression from the list.

  #include_lines: ['^ERR', '^WARN']

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that

  # are matching any regular expression from the list. By default, no files are dropped.

  #exclude_files: ['.gz$']

  # Optional additional fields. These fields can be freely picked

  # to add additional information to the crawled log files for filtering

  #fields:

  # level: debug

  # review: 1

  ### Multiline options

  # Multiline can be used for log messages spanning multiple lines. This is common

  # for Java Stack Traces or C-Line Continuation

  # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [

  #multiline.pattern: ^\[

  # Defines if the pattern set under pattern should be negated or not. Default is false.

  #multiline.negate: false

  # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern

  # that was (not) matched before or after or as long as a pattern is not matched based on negate.

  # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash

  #multiline.match: after

#============================= Filebeat modules ===============================

filebeat.config.modules:

  # Glob pattern for configuration loading

path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading

reload.enabled: false

Here is the result of ls /etc/filebeat/modules.d:

apache2.yml.disabled  auditd.yml.disabled         haproxy.yml.disabled  iis.yml.disabled    kibana.yml.disabled    mongodb.yml.disabled  nginx.yml           osquery.yml.disabled     redis.yml.disabled     system.yml           traefik.yml.disabled
auditd.yml            elasticsearch.yml.disabled  icinga.yml.disabled   kafka.yml.disabled  logstash.yml.disabled  mysql.yml.disabled    nginx.yml.disabled  postgresql.yml.disabled  suricata.yml.disabled  system.yml.disabled

Here is the result of filebeat modules list:

Enabled:
auditd
nginx
system

Disabled:
apache2
auditd
elasticsearch
haproxy
icinga
iis
kafka
kibana
logstash
mongodb
mysql
nginx
osquery
postgresql
redis
suricata
system
traefik

pierhugues · November 22, 2018, 1:42pm

Hello @princeOfMacedon,

I think we should provide a better message on our side. But I think what is going on is the following.

I think Filebeat is trying to watch the same file with two different inputs (module uses inputs).

Looking at what you have enabled in the system module, the module watch theses files:


      - /var/log/messages*
      - /var/log/syslog*
      - /var/log/auth.log*
      - /var/log/secure*

The input you have defined is using a wilcards that also watch the same as above.

- /var/log/*.log

You have to configure your inputs to exclude the files from the system module.

Something like this.

filebeat.inputs:
- type: log
  ...
  exclude_files: ["messages\.log$", "syslog$", "auth\.log$", "secure\.log$"]

NOTES: I am nore sure about the message.log and syslog you have to verify that on your system.

princeOfMacedon · November 26, 2018, 7:12am

I'm testing this but I think the regex should be encapsulated in single quote, not double quote.

However, from my testing, a fresh installation of filebeat 6.5.1 does not have this issue despite the same config. The problem only seems to happen when I update the apt package.

pierhugues · November 26, 2018, 1:17pm

single or double quote should not matter for that, because it get normalized to a string internally and transformer to a regular expression later.

Concerning the update using APT package, it should not change anything.
That code has not been change for quite some time.

princeOfMacedon · November 27, 2018, 8:00am

Thank you for your support exclude_files: ['auth\.log$'] fix this for me.
However, the problem is really happened on server where I upgrade from 6.5.0 to 6.5.1 (automatically).
New filebeat deployment on brand new server experience no problem even without any exclude_files.

system · December 25, 2018, 8:00am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.