Filebeat configuration module path issue

[tkkchan]

Dear All,
I am trying to follow this tutorial here to build a dashboard on some public data sets.

However, whenever I tried to run ./filebeat, the following error appeared:

2021-03-31T14:19:44.685+0800 ERROR [modules] fileset/modules.go:131 Not loading modules. Module directory not found: /usr/share/filebeat/bin/module

And I have found this post here

[Filebeat] Changing filebeat's path for it's configuration module

Good day @ChrsMark, I followed the installation. I figured out that the path.home was set to "/usr/share/filebeat/bin/" instead of "/usr/share/filebeat/" where the "module" directory could be found. I fixed this issue by setting the paths in filebeat's configuration file "/etc/filebeat/filebeat.yml" right after the first comments in the document. path.home : /usr/share/filebeat path.config : /etc/filebeat path.data : /var/lib/filebeat path.logs : /var/log/filebeat I installed filebeat us…

I see the same info popping up in the terminal, meaning that the path are not configured correctly.

2020-06-08T17:49:34.832-0500 INFO instance/beat.go:622 Home path: [/usr/share/filebeat/bin] Config path: [/etc/filebeat] Data path: [/usr/share/filebeat/bin/data] Logs path: [/usr/share/filebeat/bin/logs]

I tried modifying /etc/filebeat/filebeat.yml as stated in here, but it doesn't work.

Then I noted that there's some customized configuration in this example data set(you need to move a custom .yml (ncedc-earthquakes-filebeat.yml) to the /usr/share/filebeat/ directory). Then I tried to add the above configuration to this

    filebeat.inputs:

    #    - /var/log/*.log

    # 	-  /usr/share/filebeat
    #		-  /etc/filebeat
    #		-  /var/lib/filebeat
    #		-  /var/log/filebeat 

The following error occured:
Exiting: error loading config file: yaml: line 11: found character that cannot start any token

I am not sure what I did wrong with adding the config, as I am not sure how to do it as well. Any help would be appreciated, thank you

(If it helps: my version of ELK is as follows, all installed with RPM on a Centos 7)
Elasticsearch, Logstash, Kibana: 7.11.2
Filebeat: 7.12.0

[jsoriano (Jaime Soriano)]

Hi @tkkchan, welcome to discuss

By default filebeat uses the current directory as home path, it seems that you are running filebeat as ./filebeat from the /usr/share/filebeat/bin directory, and then it considers this directory its home path.

When installing using RPM or Debian packages a wrapper is installed in one of the default paths (in /usr/bin/filebeat). This wrapper includes flags to setup the proper paths:

#!/usr/bin/env bash

# Script to run Filebeat in foreground with the same path settings that
# the init script / systemd unit file would do.

exec /usr/share/filebeat/bin/filebeat \
  --path.home /usr/share/filebeat \
  --path.config /etc/filebeat \
  --path.data /var/lib/filebeat \
  --path.logs /var/log/filebeat \
  "$@"

In summary, try running filebeat without ./, so the wrapper is used. Alternatively, run filebeat with custom paths.

Also take into account that the tutorial you are following is created for Filebeat 6.0, you may find some differences with more recent versions.

[tkkchan]

Thank you very much for the prompt reply, Jamie. I figured it out as it was some other problem, but still, thank you very much for your help!

[jsoriano (Jaime Soriano)]

@tkkchan could you share your solution in case it helps others in the future? Thanks!

[tkkchan]

Dear Jamie,
I actually realised I forgot to run filebeat at the same time as logstash which is why there's no output. (yes that's stupid)

Btw, a brute-force method could be building a soft link between /usr/share/filebeat/bin and /usr/share/filebeat/bin/module as well. (by doing ln -s)
I tried it before and it silents the error(though, back then I haven't realised I didn't turn on Logstash so there's still no output, but I think it's worth a shot)
Cheers,
TK