Filebeat is harvesting the wrong files

[Tompson]

I'm using filebeat on a debian 10 to ship logfiles directly to elasticsearch und use kibana as the GUI to display an filter those files, as it's supposed to be. The problem is, that the configurations i make doesnt have any effect.
I set the paths that filebeat should harvest in the filebeat.input section. I even set the paths to exclude the paths it should not harvest. I've tried to set the paths in single quotes, double qoutes, no qoutes at all, nothing stops filebeat to harvest the wrong files.
Any idea what am i doing wrong?

Here the filebeat.yml:
#=========================== Filebeat inputs =============================

filebeat.inputs:

Each - is an input. Most options can be set at the input level, so

you can use different inputs for various configurations.

Below are the input specific configurations.

• type: log

  Change to true to enable this input configuration.

  enabled: true

  Paths that should be crawled and fetched. Glob based paths.

  paths:

  • /var/log/syslogfolder/folder_A/syslogfile_A.log
  • /var/log/syslogfolder/folder_B/syslogfile_B.log
  • /var/log/syslogfolder/folder_C/syslogfile_C.log
    #- c:\programdata\elasticsearch\logs*

ignore_older: 24h

exclude_files:
- /var/log/syslog
- /var/log/.log
- /var/log/.gz

#============================= Filebeat modules ===============================

filebeat.config.modules:

Glob pattern for configuration loading

path: ${path.config}/modules.d/*.yml

Set to true to enable config reloading

#reload.enabled: false

Period on which files under path should be checked for changes

#reload.period: 10s

#==================== Elasticsearch template setting ==========================

setup.template.settings:
index.number_of_shards: 1
#index.codec: best_compression
#_source.enabled: false

#================================ Outputs =====================================

Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:

Array of hosts to connect to.

hosts: ["localhost:9200"]

processors:

• add_host_metadata: ~
• add_cloud_metadata: ~

[pmercado (PM)]

Hi @Tompson,

can you re-post the configuration using the pre-formatted text style to check indenting?
I'm not sure how files should appear at each section, but have seen it without quotes and double quoted before.

Can also add which files are being logged that shouldn't and which are being not logged but should?

[Tompson]

I see the headline was a wrong idea, sorry for that.
Here is the configuration:

#=========================== Filebeat inputs =============================
filebeat.inputs:
- type: log

paths:
- /var/log/syslogfolder/folder_A/syslogfile_A.log
- /var/log/syslogfolder/folder_B/syslogfile_B.log
- /var/log/syslogfolder/folder_C/syslogfile_C.log

exclude_files:
- /var/log/syslog
- /var/log/*.log
- /var/log/*.gz
#============================= Filebeat modules ===============================
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
#==================== Elasticsearch template setting ==========================
setup.template.settings:
  index.number_of_shards: 1
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  hosts: ["localhost:9200"]
#================================ Processors =====================================
    processors:
      - add_host_metadata: ~
      - add_cloud_metadata: ~

All files at the /var/log/* folder and subfolders being harvested. But, as i configured in the filebeat.yml, i want filebeat only to harvest the files configured under the inputs-section and exclude the files under the exclude_files-section (which would not be necessary, if filebeat would ignor all files except the files being set at the input). And it seems, that changes in the filebeat.yml have no effect at all.
Any idea whats wrong?

[kumarabhi (Abhishek)]

You don't need to exclude any path if you have explicitly provided full file path in the input section.
It should be used in case you use a file pattern for input and want to exclude some files.

filebeat.inputs:
 - type: log

   paths:
      - /var/log/syslogfolder/folder_A/syslogfile_A.log
      - /var/log/syslogfolder/folder_B/syslogfile_B.log
      - /var/log/syslogfolder/folder_C/syslogfile_C.log

Just make sure that you are providing the correct configuration file (filebeat.yml)
Start filebeat using command line with -c option

Ex : /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml (replace with settings on your local machine)