Multiple definitions for the same module?

[sebastien.k (Sébastien Kurtzemann)]

Hi,

We have created a specific filebeat module to parse Drupal logs.

In filebeat.yml we declare two entries for filebeat.modules with this custom module in order to distinguish preprod and prod environments (by adding some fields and tags).

Here the config

filebeat.modules:
- module: drupal
  syslog:
    enabled: true
    var.paths:
      - /home/client/preprod.client.fr/log/drupal/*.log
    prospector:
      fields:
        project_customer: "client"
        project_name: "preprod.client.fr"
        origin: "drupal-syslog"
      tags:
        - "preprod.client.fr"
        - "website"

- module: drupal
  syslog:
    enabled: true
    var.paths:
      - /home/client/www.client.fr/log/drupal/*.log
    prospector:
      fields:
        project_customer: "client"
        project_name: "www.client.fr"
        origin: "drupal-syslog"
      tags:
        - "www.client.fr"
        - "website"

But when filebeat start (version 5.6.0), only the last entry is loaded

2017/11/07 15:52:29.453488 log.go:91: INFO Harvester started for file: /home/client/www.client.fr/log/drupal/drupal.log

I have made some tests

• disabling the second entry (enabled: false) : this made the two modules definitions disabled and we got an error

  2017/11/07 15:48:53.973675 beat.go:346: CRIT Exiting: No modules or prospectors enabled and configuration reloading disabled. What files do you want me to watch?
  

• remove the second entry : the first definition is loaded

  2017/11/07 15:50:48.751536 log.go:91: INFO Harvester started for file: /home/client/preprod.client.fr/log/drupal/drupal.log
  

My questions are

• Multiple definitions for the same module aren't possible ?
• Do I use the correct method to do such things ?

Thanks,
Sébastien KURTZEMANN

[Vitaly_il (Vitaly)]

@sebastien.k

filebeat.modules

I guess you should use "drupal_prod" and "drupal_preprod" as module names.

[sebastien.k (Sébastien Kurtzemann)]

@Vitaly_il : but if I do that I need to duplicate the "drupal" module code, no ?

We want to have one drupal module and have multiples declarations for it

[Vitaly_il (Vitaly)]

Well, I may be wrong...
Let's wait for Elastic team answer

[kvch (Noémi Ványi)]

Hi!

When your configuration is normalized the first drupal module gets overwritten by the second one.

AFAIK right now it is not supported to have multiple definitions for the same module by Filebeat.
But I think @tudor can add more on this topic.

[sebastien.k (Sébastien Kurtzemann)]

Thanks for this explanation @kvch !

I'll wait for more informations from @tudor