[Filebeat] Changing filebeat's path for it's configuration module

[gonzalo2kx (Gonzalo De La Torre Parra)]

Good day,

I am currently experiencing a problem to load the system module on filebeat. The system module has been enabled and verified using "filebeat modules list". When I launch filebeat 7.6.2 in publish mode I get the following error:

"Not loading modules. Module directory not found: /usr/share/filebeat/bin/module"

During launch time, the first line I get is the following:

2020-06-08T17:34:49.831-0500	INFO	instance/beat.go:622	Home path: [/usr/share/filebeat/bin] Config path: [/usr/share/filebeat/bin] Data path: [/usr/share/filebeat/bin/data] Logs path: [/usr/share/filebeat/bin/logs]

As far as I understand, whenever we enable a module within filebeat, the suffix from the file "system.yml.disabled" changes to "system.yml". I have validated this but contrary from where filebeat expect to observe this file " /usr/share/filebeat/bin/module" , all module yml files are actually located in "/etc/filebeat/modules.d/".

I used two methods in my attempt to fix this issue:

1. Enable and change the module's configuration path in "/etc/filebeat/filebeat.yml":

filebeat.config.modules:
  enabled: true
  path: /etc/filebeat/modules.d/*.yml

2. Change the configuration path at the start of filebeat's configuration path in "/etc/filebeat/filebeat.yml" by stating:

path.config: /etc/filebeat

Followed by stating the module's configuration path as follows:

filebeat.config.modules:
  enabled: true
  path: ${path.config}/modules.d/*.yml

This resulted in the following Config path change on runtime:

2020-06-08T17:49:34.832-0500	INFO	instance/beat.go:622	Home path: [/usr/share/filebeat/bin] Config path: [/etc/filebeat] Data path: [/usr/share/filebeat/bin/data] Logs path: [/usr/share/filebeat/bin/logs]

Both approaches resulted in the same error presented below. Tried to restart the system to see any difference but no luck.

2020-06-08T17:49:35.879-0500	ERROR	fileset/modules.go:125	Not loading modules. Module directory not found: /usr/share/filebeat/bin/module

Please help

[ChrsMark (Chris Mark)]

Hi!

How exactly do you install Filebeat?

Did you try to follow exactly the process that is described in the docs (https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-installation.html) and check if works?

C.

[gonzalo2kx (Gonzalo De La Torre Parra)]

Good day @ChrsMark,

I followed the installation. I figured out that the path.home was set to "/usr/share/filebeat/bin/" instead of "/usr/share/filebeat/" where the "module" directory could be found. I fixed this issue by setting the paths in filebeat's configuration file "/etc/filebeat/filebeat.yml" right after the first comments in the document.

path.home : /usr/share/filebeat
path.config : /etc/filebeat
path.data : /var/lib/filebeat
path.logs : /var/log/filebeat

I installed filebeat using the following commands:

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.7.1-amd64.deb
sudo dpkg -i filebeat-7.7.1-amd64.deb

This seems to be a bug that in some installations appears and in some others disappears even when using the sam cloud image.

Another issue I have encountered is that the modules configuration path is changed from "/etc/filebeat". Therefore, to solve this issue I need to set "path.config : /etc/filebeat", as stated on the comments above, along with the following changes in filebeat's configuration:

From this configuration:

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

To the following specific path:

filebeat.config.modules:
  path : /etc/filebeat/modules.d/*.yml

I hope this helps other users going through the same struggles. This has been a recurring issue that has not been properly addressed.

Notes:

1. The directory "/etc/filebeat/modules.d/" contains a list of yml files that state which modules are enabled and disabled in filebeat.
2. The directory "/usr/share/filebeat/module/" contains subdirectories named after each available module (enabled or disabled). Each subdirectory contains the following:
   a) A file named "module.yml" presenting a list of dashboards associated to the module and the corresponding IDs
   b) A set of sub-sub-directories associated with the fileset/dataset of the corresponding module (e.g, auth and syslog for the system module). Each of these contain 3 additional yml files; one for configuration, one to set the pipeline when connecting filebeat directly to elasticsearch, and one manifest file.