Filebeat startup errors and source issue

[Cdnvballer (Christian Vo)]

current state:
elk 6.8.5 setup, which seems to be parsing and visualizing... however:

/etc/filebeat/filebeat.yml has the following ;
I only want filebeat to look at 1 logfile (/var/log/network):
"- type: log
- /var/log/network
exclude_files: ['.gz$']
"
/etc/filebeat/modules.d/system.yml has the following: "
"- module: system
syslog:
enabled: true
var.paths: ["/var/log/network"]
"

Issues I'm seeing:

1. when I restart filebeat, I keep seeing this error:
   ERROR instance/beat.go:906 Exiting: Can only start an input when all related states are finished: {Id:4458412-64512 Finished:false Fileinfo:0xc420582680 Source:/var/log/network Offset:0 Timestamp:2019-12-02 12:28:40.849396001 -0800 PST m=+0.049508312 TTL:-1ns Type:log Meta:map FileStateOS:4458412-64512}

I've tried removing the entry for "source":"/var/log/network"... from /var/lib/filebeat/registry but error still repeats...

2. when I look at Kibana discover console, [fileset][name] is syslog as expected
   but [source] is "/var/log/syslog" instead of "/var/log/network "

how do I fix these issues? I don't recall anywhere else to specify/limit the logfile to be checked by filebeat

[Cdnvballer (Christian Vo)]

has something to do with /etc/filebeat/modules.d/system.yml

if I leave this commented, filebeats starts up

#var.paths: [/var/log/network]

however it starts looking at these instead of just the single file

2019-12-02T16:18:12.637-0800 INFO log/input.go:148 Configured paths: [/var/log/messages* /var/log/syslog*]
2019-12-02T16:18:12.637-0800 INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 1
2019-12-02T16:18:12.637-0800 INFO cfgfile/reload.go:150 Config reloader started
2019-12-02T16:18:12.640-0800 INFO log/input.go:148 Configured paths: [/var/log/auth.log* /var/log/secure*]
2019-12-02T16:18:12.643-0800 INFO log/input.go:148 Configured paths: [/var/log/messages* /var/log/syslog*]
2019-12-02T16:18:12.643-0800 INFO input/input.go:114 Starting input of type: log; ID: 11691604117334136852
2019-12-02T16:18:12.643-0800 INFO input/input.go:114 Starting input of type: log; ID: 15959826002125184592
2019-12-02T16:18:12.643-0800 INFO cfgfile/reload.go:205 Loading of config files completed.
2019-12-02T16:18:12.644-0800 INFO log/harvester.go:255 Harvester started for file: /var/log/syslog
2019-12-02T16:18:12.644-0800 INFO log/harvester.go:255 Harvester started for file: /var/log/auth.log

[Cdnvballer (Christian Vo)]

alright figured it out...

I can't use /var/log/network in BOTH filebeat.yml and system.yml

solution was to revert the entry in filebeat.yml as follows:
"filebeat.inputs:

• type: log
  paths:
  • "/var/log/*.log"
    "

while keeping /etc/filebeat/modules.d/system.yml as follows:
" syslog:
enabled: true

# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
var.paths: ["/var/log/network"]

"

once I did this, filebeats started properly and used /var/log/network as expected...