New filebeat module - how to move to production? Template not loading

Elastic Stack Beats
,
1

I have created a new module to Filebeat. To be honest I have been struggling with this for a long time, perhaps a blog post somewhere (creating a new filebeat start to finish) I could find could help me immensely. Anyway. I have cloned the beats repository, created a filebeat module, and gone through and tested it. This module has new fields which I need loaded into the template, and here is where I cannot connect the dots. I have run "make update" which updates the fields.yml file at the top level beats/Filebeat dev folder.

How do I get this module into production?

I have and installed filebeat module on Linux, and have replaced the fields.yml from the dev folder (which has the additional fields included in it) into /etc/filebeat folder. I then delete the template on Elastic and start filebeat. It creates the template again, but not with the additional fields I need.

so, questions
a) How do I get the additional fields loaded into the template
b) Is there a way to have a per module template being built at runtime. My understanding is that if the filebeat module is upgraded, it may overwrite fields.yml and donk my custom fields? Is this true - if so, surely there has to be a better way?
c) What is the method to move from Dev to Prod for a filebeat module

2

Are you trying to build from source? Or just add a module to an existing install of Filebeat?

3

Currently I have pulled the beats github respository, for the development, then tried to add the module to an existing install.

Not sure of what I am really supposed to do, as I cannot see anything that says one way or another.

4

If u build from source create the new module and Fileset. Then run mage update and then run mage build from within the filebeat directory.

5

Thanks for the response, but I am still missing one bit. Once I have mage update and mage build which I have done, that gets it sorted. what do I need to copy over to a production instance to get the module work, and the template loaded?

I was thinking that I would develop module in a 'dev' environment with the githib module. Then, then tested, move the relevant bits over to a production install of filebeat. Is that OK thinking, or not quite right. This is the bit that I am missing.

6

Gotcha. You want to know how to package it up into a tar, dev, rpm.... that u can then install onto a system. That's a great question that I don't have an answer for you.

7

Exactly. Although I dont necessarily need it into an RPM/deb, as then whenever the main package gets updated I would need to go through the steps again. I am hoping that there is a decoupling of the main 'engine', that is available filebeat package, and custom modules/filesets that people create. Even steps of copy modules/module.yml.disabled to ... Copy modules.d/module to ... Copy fields.yml to ...., copy ... to ....

At the moment I am stuck with a module and cannot put it into production.

8

OK - I have figured it out. To load up the modified fields.yml file into the template, you need to run the filebeat module that is generated, as the index template is part of the filebeat module (why is that not separate to the main executable!) and does not use the fields.yml file at all. I also could not find the filebeat.template.yml file that is mentioned elsewere in the discuss forum so I gather this was deprecated in v6 or v7 beats? So, my understanding, to create a filebeat module, which has fields not already defined in the template (ie you have custom fields), requires that you compile a separate standalone instance of Filebeat and potentially run that, or use it to export the template. That to me is just ridiculous. Filebeat should be the engine, and the template it loads, same as the modules should be separate to the engine and not part of it.

So, from what I read here: Where is the default template.json file for filebeat - #3 by Daud_Ahmed you need to export the template - and load it up into Elastic, and then rinse/repeat for each filebeat module that you deploy to the environment in the future that requires those additional fields. This sounds like a nightmare if you have different modules across the environment. I gather component templates might fix this - haven't got my head into this.

Can someone update documents somewhere about this (it may be there but I did not come across it)

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.