Getting custom filebeat module into production

[kelvins]

I have created a new beats module (jde) which contain a couple of filesets. I have gone through the testing phases of the filebeat modules and all tests pass - Yeah!

I am now trying to move this into production. To do this, I have copied my module onto a production server in the modules folder. I have created a /etc/filebeat/modules.d/jde.yml which contains

# Module: jde
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.10/filebeat-module-nginx.html

- module: jde
  # Access logs
  kernel:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths:
      - /home/kelvins/Oracle/Middleware/user_projects/domains/E1_Appsservers/PP129_J2EE_SI_BSSVlogs/jde*.log

  bssi_in:
    enabled: true
    var.paths:
      - /home/kelvins/Oracle/Middleware/user_projects/domains/E1_Appsservers/PP129_J2EE_SI_BSSVlogs/*_BSSV.log*

    input:
      multiline.pattern: '^###'
      multiline.negate: true
      multiline.match: after

This module contains a number customized field mappings. I can see this reflected in the fields.yml file on my dev machine. I have copied this file (which contains the jde definitions) onto my prod machine on /etc/filebeat/fields.yml.

I then run filebeat setup and am expecting that the filebeat index will be updated - it isn't. So I delete the filebeat index and try again. It creates the filebeat index, but does not contain the jde definitions.

What am I doing wrong? What is the best way to moving a module from Dev into Production and therefore the upkeep of the module as filebeat agent gets updated? I gather and update will overwrite filebeat.yml in /etc/filebeat folder therefore removing my custom definitions.

I think I have look around for the answers, but cannot seem to find them.

[Mario_Castro]

Hi @kelvins

First think I'll do is to try to make it work in a clean environment of indices on elasticsearch and other stuff like "custom things" in Filebeat, you must ensure that there's a module that Filebeat recognizes. Just using some docker container with elasticsearch. If configuration is correct you don't even need to run filebeat setup.

So the first step is to have all your stuff in place in a clean environment, no extra stuff, just your pipeline and everything else at the bare minimum (mimic the structure of any other module as reference, like mysql module), run filebeat modules list and check if your module is there.

If your module is there, then start adding things incrementally. There are a lot of small pieces in Filebeat that needs to be placed carefully and the smallest typo can give a silent error (like an error in multiline matching).

Keep doing incrementals until you find an error and post it here.

[kelvins]

Hello Mario_Castro

I have tried to get back onto this. OK - Not really sure of everything above. I have a clean(ish) elastic instance. I delete the index template, indices and kibana index patterns - effectively wiping filebeat data from Elastic.

I then run filebeat setup -e. I can see it loading up a template, but not sure where it is getting it from. It is loading up my ingest pipeline which is great, so it must be loading in the module (a filebeat modules list is showing it as enabled).

Once the template is loaded, if I go and look at the index template mappings of Filebeat, it does not show any of my modules fields at all (jde).

If I then run filebeat, it ingests the files, but the field definitions are not what they should be. That is, I have a field definition as keyword, and ignore_above value. If i look at the mappings of the index, it is showing the jde fields, but I gather these are being added because of the ingest pipeline, and not following the template.

So, I gather that filebeat loads (on linux) the /etc/filebeat/fields.yml into Elastic as the index template. Is this correct?
This is the bit that does not seem to be working for me.

Any help would be appreciated.

[Mario_Castro]

Hi @kelvins sorry for the late reply.

Why don´t you open a PR to contribute your module to the community? It's extremely difficult to help if we cannot see your code and what might be wrong. Maybe you are lacking some make update command in the filebeat root.

Now that you know that your module is loading, I also suggest to set console as output (remove elasticsearch as output) and see if your message appears in the console as expected:

# filebeat.yml file
output.console:
    pretty: true

You have to add the fields of your module in the _meta/fields.yml of your module folder too, then run the make update I mentioned before.

[kelvins]

Mario

Thanks for getting back to me. I will santisie the test files, and then upload to GitHub for you to have a look at. Problem with the module at the moment, is some of the fields are ‘made up’ as cannot find a full spec for it.

Will get back to you in a day or so.