Error creating input: Can only start an input when all related states are finished


(Hoon Cho) #1

I run filebeat with system module enabled and filebeat log like this..

2018-11-19T07:18:31.662+0900 INFO log/input.go:138 Configured paths: [/var/log/secure*]
2018-11-19T07:18:31.666+0900 ERROR fileset/factory.go:105 Error creating input: Can only start an input when all related states are finished: {Id:17600537-64768 Finished:false Fileinfo:0xc4203f0b60 Source:/var/log/messages Offset:1950248 Timestamp:2018-11-19 07:18:26.962331439 +0900 KST m=+428.768260414 TTL:-1ns Type:log Meta:map FileStateOS:17600537-64768}
2018-11-19T07:18:31.666+0900 ERROR [reload] cfgfile/list.go:96 Error creating runner from config: Can only start an input when all related states are finished: {Id:17600537-64768 Finished:false Fileinfo:0xc4203f0b60 Source:/var/log/messages Offset:1950248 Timestamp:2018-11-19 07:18:26.962331439 +0900 KST m=+428.768260414 TTL:-1ns Type:log Meta:map FileStateOS:17600537-64768}

elasticsearch cannot parsing system log exactly.
[Filebeat System] Syslog dashboard cannot show system log and auth log.
I have many logs between 04:43 to 07:10 but dashboard cannot recognize that logs.

here is my config

/etc/filebeat/filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/secure*
    - /var/log/messages*
    - /var/log/yum.log*
    - /var/log/maillog*
    - /var/log/cron*
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 10s
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
  host: "snip"
cloud.id: starwars:snip
output.elasticsearch:
  hosts: ["snip"]
  protocol: "https"
  username: "elastic"
  password: "snip"
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

/etc/filebeat/modules.d/system.yml

- module: system
  syslog:
    enabled: true
    var.paths: ["/var/log/messages*"]
  auth:
    enabled: true
    var.paths: ["/var/log/secure*"]
    var.convert_timezone: true

anyone know why this situations?


(Pier-Hugues Pellerin) #2

@Hoon_Cho

Looking at your configuration and your error message, I think the following is happening,
your manual configuration and system module are watching the same files.

if you remove the following lines in your defined inputs it should solve the problem.

 - /var/log/secure*
- /var/log/messages*

(Hoon Cho) #3

thanks for reply

you mean that remove paths from filebeat.yml? not in system.yml

like this?

/etc/filebeat/filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/yum.log*
    - /var/log/maillog*
    - /var/log/cron*

(Pier-Hugues Pellerin) #4

@Hoon_Cho Yes exactly