# Filebeat 6.5.1 unable to send syslog

**URL:** <https://discuss.elastic.co/t/filebeat-6-5-1-unable-to-send-syslog/157819>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [November 22, 2018, 8:17am UTC](https://discuss.elastic.co/t/filebeat-6-5-1-unable-to-send-syslog/157819 "2018-11-22T08:17:10Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![princeOfMacedon](https://avatars.discourse-cdn.com/v4/letter/p/f04885/32.png) [@princeOfMacedon](https://discuss.elastic.co/u/princeOfMacedon)\
**Post date:** [November 22, 2018, 8:17am UTC](https://discuss.elastic.co/t/filebeat-6-5-1-unable-to-send-syslog/157819/1 "2018-11-22T08:17:10Z")

</div>

Hi all!

I'm using filebeat 6.5.1 and I'm encountering this error:

filebeat[29062]: Exiting: Can only start an input when all related states are finished: {Id:2537-66305 Finished:false Fileinfo:0xc420213860 Source:/var/log/auth.log Offset:618294 Timestamp:2018-11-22 07:10:05.795725847 +0000 UTC m=+0.072986119 TTL:-1ns Type:log Meta:map FileS

I follow the instruction here [Filebeat: Exiting: Can only start an input when all related states are finished](https://discuss.elastic.co/t/filebeat-exiting-can-only-start-an-input-when-all-related-states-are-finished/147692) and delete /var/lib/filebeat/registry.

Before deleting, the registry content is:

[{"source":"/var/log/cloud-init-output.log","offset":4850,"timestamp":"2018-11-13T05:21:23.36633439Z","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":2974,"device":66305}},{"source":"/var/log/cloud-init.log","offset":147844,"timestamp":"2018-11-13T05:21:23.400354561Z","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":2971,"device":66305}},{"source":"/var/log/dpkg.log","offset":152397,"timestamp":"2018-11-21T02:59:00.553800623Z","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":58290,"device":66305}},{"source":"/var/log/kern.log","offset":199276,"timestamp":"2018-11-22T07:04:14.668430496Z","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":17629,"device":66305}},{"source":"/var/log/alternatives.log","offset":528,"timestamp":"2018-11-13T05:21:48.366825045Z","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":2328,"device":66305}},{"source":"/var/log/auth.log","offset":694771,"timestamp":"2018-11-22T07:22:00.6795606Z","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":17713,"device":66305}},{"source":"/var/log/syslog","offset":22833471,"timestamp":"2018-11-22T07:25:39.441186992Z","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":2366,"device":66305}},{"source":"/var/log/syslog.1","offset":525329504,"timestamp":"2018-11-22T06:30:17.939994178Z","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":2358,"device":66305}},{"source":"/var/log/syslog.2","offset":196173115,"timestamp":"2018-11-22T06:54:22.880009798Z","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":2309,"device":66305}}]

After that, I restart filebeat, it show no error, but no syslog is pushed to elastic while nginx log still works as expected.

The registry content is now:

[{"source":"/var/log/dpkg.log","offset":152397,"timestamp":"2018-11-22T07:52:15.145040968Z","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":58290,"device":66305}},{"source":"/var/log/kern.log","offset":199276,"timestamp":"2018-11-22T07:52:18.005431008Z","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":17629,"device":66305}},{"source":"/var/log/alternatives.log","offset":528,"timestamp":"2018-11-22T07:52:13.698411707Z","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":2328,"device":66305}},{"source":"/var/log/auth.log","offset":703210,"timestamp":"2018-11-22T07:52:30.223743183Z","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":17713,"device":66305}},{"source":"/var/log/cloud-init-output.log","offset":4850,"timestamp":"2018-11-22T07:52:13.763932619Z","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":2974,"device":66305}},{"source":"/var/log/cloud-init.log","offset":147844,"timestamp":"2018-11-22T07:52:15.912014197Z","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":2971,"device":66305}},{"source":"/var/log/nginx/access.log","offset":45367,"timestamp":"2018-11-22T07:55:02.519360467Z","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":515336,"device":66305}},{"source":"/var/log/nginx/access.log.1","offset":781856,"timestamp":"2018-11-22T07:52:21.674422476Z","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":516014,"device":66305}},{"source":"/var/log/nginx/error.log","offset":0,"timestamp":"2018-11-22T07:52:13.686014045Z","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":515668,"device":66305}},{"source":"/var/log/nginx/error.log.1","offset":4648,"timestamp":"2018-11-22T07:52:15.205459069Z","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":516326,"device":66305}}]

Thanks in advanced!

---

<div class="post-metadata">

**Author:** ![princeOfMacedon](https://avatars.discourse-cdn.com/v4/letter/p/f04885/32.png) [@princeOfMacedon](https://discuss.elastic.co/u/princeOfMacedon)\
**Post date:** [November 22, 2018, 8:21am UTC](https://discuss.elastic.co/t/filebeat-6-5-1-unable-to-send-syslog/157819/2 "2018-11-22T08:21:46Z")

</div>

Here is my /etc/filebeat/module.d/system.yml

```auto
- module: system
  # Syslog
  syslog:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

    # Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
    var.convert_timezone: true

  # Authorization logs
  auth:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

    # Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
    var.convert_timezone: true

```

Here is a part of my /etc/filebeat/filebeat.yml:

```auto
- type: log

  # Change to true to enable this input configuration.

enabled: true

  # Paths that should be crawled and fetched. Glob based paths.

paths:

- /var/log/*.log

#- c:\programdata\elasticsearch\logs\*

  # Exclude lines. A list of regular expressions to match. It drops the lines that are

  # matching any regular expression from the list.

  #exclude_lines: ['^DBG']

  # Include lines. A list of regular expressions to match. It exports the lines that are

  # matching any regular expression from the list.

  #include_lines: ['^ERR', '^WARN']

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that

  # are matching any regular expression from the list. By default, no files are dropped.

  #exclude_files: ['.gz$']

  # Optional additional fields. These fields can be freely picked

  # to add additional information to the crawled log files for filtering

  #fields:

  # level: debug

  # review: 1

  ### Multiline options

  # Multiline can be used for log messages spanning multiple lines. This is common

  # for Java Stack Traces or C-Line Continuation

  # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [

  #multiline.pattern: ^\[

  # Defines if the pattern set under pattern should be negated or not. Default is false.

  #multiline.negate: false

  # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern

  # that was (not) matched before or after or as long as a pattern is not matched based on negate.

  # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash

  #multiline.match: after

#============================= Filebeat modules ===============================

filebeat.config.modules:

  # Glob pattern for configuration loading

path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading

reload.enabled: false

```

Here is the result of ls /etc/filebeat/modules.d:

```auto
apache2.yml.disabled auditd.yml.disabled haproxy.yml.disabled iis.yml.disabled kibana.yml.disabled mongodb.yml.disabled nginx.yml osquery.yml.disabled redis.yml.disabled system.yml traefik.yml.disabled
auditd.yml elasticsearch.yml.disabled icinga.yml.disabled kafka.yml.disabled logstash.yml.disabled mysql.yml.disabled nginx.yml.disabled postgresql.yml.disabled suricata.yml.disabled system.yml.disabled

```

Here is the result of filebeat modules list:

```auto
Enabled:
auditd
nginx
system

Disabled:
apache2
auditd
elasticsearch
haproxy
icinga
iis
kafka
kibana
logstash
mongodb
mysql
nginx
osquery
postgresql
redis
suricata
system
traefik

```

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [November 22, 2018, 1:42pm UTC](https://discuss.elastic.co/t/filebeat-6-5-1-unable-to-send-syslog/157819/3 "2018-11-22T13:42:38Z")

</div>

Hello @princeOfMacedon,

I think we should provide a better message on our side. But I think what is going on is the following.

**I think Filebeat is trying to watch the same file with two different inputs (module uses inputs).**

Looking at what you have enabled in the system module, the module watch theses files:

```auto

      - /var/log/messages*
      - /var/log/syslog*
      - /var/log/auth.log*
      - /var/log/secure*

```

The input you have defined is using a wilcards that also watch the same as above.

```auto
- /var/log/*.log

```

You have to configure your inputs to [exclude the files](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html#filebeat-input-log-exclude-files) from the system module.

Something like this.

```auto
filebeat.inputs:
- type: log
  ...
  exclude_files: ["messages\.log$", "syslog$", "auth\.log$", "secure\.log$"]

```

**NOTES:** I am nore sure about the **message.log** and **syslog** you have to verify that on your system. 🙂

---

<div class="post-metadata">

**Author:** ![princeOfMacedon](https://avatars.discourse-cdn.com/v4/letter/p/f04885/32.png) [@princeOfMacedon](https://discuss.elastic.co/u/princeOfMacedon)\
**Post date:** [November 26, 2018, 7:12am UTC](https://discuss.elastic.co/t/filebeat-6-5-1-unable-to-send-syslog/157819/4 "2018-11-26T07:12:42Z")

</div>

I'm testing this but I think the regex should be encapsulated in single quote, not double quote.

However, from my testing, a fresh installation of filebeat 6.5.1 does not have this issue despite the same config. The problem only seems to happen when I update the apt package.

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [November 26, 2018, 1:17pm UTC](https://discuss.elastic.co/t/filebeat-6-5-1-unable-to-send-syslog/157819/5 "2018-11-26T13:17:09Z")

</div>

single or double quote should not matter for that, because it get normalized to a string internally and transformer to a regular expression later.

Concerning the update using APT package, it should not change anything.  
That code has not been change for quite some time.

---

<div class="post-metadata">

**Author:** ![princeOfMacedon](https://avatars.discourse-cdn.com/v4/letter/p/f04885/32.png) [@princeOfMacedon](https://discuss.elastic.co/u/princeOfMacedon)\
**Post date:** [November 27, 2018, 8:00am UTC](https://discuss.elastic.co/t/filebeat-6-5-1-unable-to-send-syslog/157819/6 "2018-11-27T08:00:31Z")

</div>

Thank you for your support `exclude_files: ['auth\.log$']` fix this for me.  
However, the problem is really happened on server where I upgrade from 6.5.0 to 6.5.1 (automatically).  
New filebeat deployment on brand new server experience no problem even without any exclude\_files.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 25, 2018, 8:00am UTC](https://discuss.elastic.co/t/filebeat-6-5-1-unable-to-send-syslog/157819/7 "2018-12-25T08:00:34Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.