Only first path being harvested

[jrubenc (Ruben Cardenal)]

Hello,

I have the following simple configuration:

# cat /etc/filebeat/filebeat.yml

# Workaround manually managed

################### filebeat Configuration #########################

############################# filebeat ######################################
filebeat:
  config:
    modules:
      path: /usr/share/filebeat/module
      reload:
        enabled: false
  modules:
  - auth:
      enabled: false
    module: system
    syslog:
      enabled: false
  - server:
      enabled: true
      var:
        paths:
        - /var/log/elasticsearch/localhost-node/hidden-filename-1.log
        - /var/log/elasticsearch/localhost-node/hidden-filename-2.log
    module: elasticsearch
path:
  config: /etc/filebeat
  data: /var/lib/filebeat
  home: /usr/share/filebeat
  logs: /var/log/filebeat
processors:
- add_host_metadata: null
- add_cloud_metadata: null


###############################################################################
############################# Libbeat Config ##################################
# Base config file used by all other beats for using libbeat features

############################# Output ##########################################

output:
  kafka:
    compression: gzip
    hosts:
    - monitoring.hidden.com:9093
    - monitoring.hidden.com:9094
    - monitoring.hidden.com:9095
    max_message_bytes: 1000000
    password: -----------
    required_acks: '-1'
    ssl:
      enabled: true
    topic: kafka.topic.name
    topics:
    - topic: kafka.topic.name
      when:
        equals:
          event:
            module: system
    - topic: kafka.topic.name
      when:
        equals:
          event:
            module: elasticsearch
    username: producer


############################# Logging #########################################

logging:
  files:
    keepfiles: 7
    name: filebeat
    path: /var/log/filebeat
    permissions: 420

Whenever a new line is added to /var/log/elasticsearch/localhost-node/hidden-filename-1.log you can see immediately in syslog the corresponding:

Harvester started for file: /var/log/elasticsearch/localhost-node/hidden-filename-1.log

But when a new line is added to /var/log/elasticsearch/localhost-node/hidden-filename-2.log nothing happens, and its contents are not exported.

What's wrong here?

If I swap the paths, as in:

        - /var/log/elasticsearch/localhost-node/hidden-filename-2.log
        - /var/log/elasticsearch/localhost-node/hidden-filename-1.log

then ONLY "filename-1" is taken into account, as it happened before.

Both files have the same permissions:

-rw-r--r-- 1 elasticsearch elasticsearch  298 Nov 21 08:33 hidden-filename-1.log
-rw-r--r-- 1 elasticsearch elasticsearch  204 Nov 21 08:42 hidden-filename-2.log

I can see for example here How to configure apache.yml in filebeat with multiple paths in var.paths that the configuration seems to be correct, but I don't know if I'm missing something here.

Thanks.

[jrubenc (Ruben Cardenal)]

Moreover, if I leave only

    - /var/log/elasticsearch/localhost-node/hidden-filename-2.log

it's not taken into account at all.

So I really don't know what's the problem here. Any help would be appreciated.

Thanks!