# Getting custom filebeat module into production **URL:** \ **Category:** Beats\ **Tags:** beats-development\ **Created:** [June 12, 2021, 9:41pm UTC](https://discuss.elastic.co/t/getting-custom-filebeat-module-into-production/275740 "2021-06-12T21:41:47Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![kelvins](https://avatars.discourse-cdn.com/v4/letter/k/f19dbf/32.png) [@kelvins](https://discuss.elastic.co/u/kelvins)\ **Post date:** [June 12, 2021, 9:41pm UTC](https://discuss.elastic.co/t/getting-custom-filebeat-module-into-production/275740/1 "2021-06-12T21:41:47Z")
I have created a new beats module (jde) which contain a couple of filesets. I have gone through the testing phases of the filebeat modules and all tests pass - Yeah! I am now trying to move this into production. To do this, I have copied my module onto a production server in the modules folder. I have created a /etc/filebeat/modules.d/jde.yml which contains ```auto # Module: jde # Docs: https://www.elastic.co/guide/en/beats/filebeat/7.10/filebeat-module-nginx.html - module: jde # Access logs kernel: enabled: true # Set custom paths for the log files. If left empty, # Filebeat will choose the paths depending on your OS. var.paths: - /home/kelvins/Oracle/Middleware/user_projects/domains/E1_Appsservers/PP129_J2EE_SI_BSSVlogs/jde*.log bssi_in: enabled: true var.paths: - /home/kelvins/Oracle/Middleware/user_projects/domains/E1_Appsservers/PP129_J2EE_SI_BSSVlogs/*_BSSV.log* input: multiline.pattern: '^###' multiline.negate: true multiline.match: after ``` This module contains a number customized field mappings. I can see this reflected in the fields.yml file on my dev machine. I have copied this file (which contains the jde definitions) onto my prod machine on /etc/filebeat/fields.yml. I then run `filebeat setup` and am expecting that the filebeat index will be updated - it isn't. So I delete the filebeat index and try again. It creates the filebeat index, but does not contain the jde definitions. What am I doing wrong? What is the best way to moving a module from Dev into Production and therefore the upkeep of the module as filebeat agent gets updated? I gather and update will overwrite filebeat.yml in /etc/filebeat folder therefore removing my custom definitions. I think I have look around for the answers, but cannot seem to find them. ---
**Author:** ![Mario\_Castro](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mario_castro/32/35107_2.png) [@Mario\_Castro](https://discuss.elastic.co/u/Mario_Castro)\ **Post date:** [June 14, 2021, 9:01am UTC](https://discuss.elastic.co/t/getting-custom-filebeat-module-into-production/275740/2 "2021-06-14T09:01:23Z")
Hi @kelvins 🙂 First think I'll do is to try to make it work in a clean environment of indices on elasticsearch and other stuff like "custom things" in Filebeat, you must ensure that there's a module that Filebeat recognizes. Just using some docker container with elasticsearch. If configuration is correct you don't even need to run `filebeat setup`. So the first step is to have all your stuff in place in a clean environment, no extra stuff, just your pipeline and everything else at the bare minimum (mimic the structure of any other module as reference, like `mysql` module), run `filebeat modules list` and check if your module is there. If your module is there, then start adding things incrementally. There are a lot of small pieces in Filebeat that needs to be placed carefully and the smallest typo can give a silent error (like an error in multiline matching). Keep doing incrementals until you find an error and post it here. ---
**Author:** ![kelvins](https://avatars.discourse-cdn.com/v4/letter/k/f19dbf/32.png) [@kelvins](https://discuss.elastic.co/u/kelvins)\ **Post date:** [June 17, 2021, 10:37am UTC](https://discuss.elastic.co/t/getting-custom-filebeat-module-into-production/275740/3 "2021-06-17T10:37:44Z")
Hello Mario\_Castro I have tried to get back onto this. OK - Not really sure of everything above. I have a clean(ish) elastic instance. I delete the index template, indices and kibana index patterns - effectively wiping filebeat data from Elastic. I then run `filebeat setup -e`. I can see it loading up a template, but not sure where it is getting it from. It is loading up my ingest pipeline which is great, so it must be loading in the module (a `filebeat modules list` is showing it as enabled). Once the template is loaded, if I go and look at the index template mappings of Filebeat, it does not show any of my modules fields at all (jde). If I then run filebeat, it ingests the files, but the field definitions are not what they should be. That is, I have a field definition as keyword, and `ignore_above` value. If i look at the mappings of the index, it is showing the jde fields, but I gather these are being added because of the ingest pipeline, and not following the template. So, I gather that filebeat loads (on linux) the `/etc/filebeat/fields.yml` into Elastic as the index template. Is this correct? This is the bit that does not seem to be working for me. Any help would be appreciated. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 15, 2021, 12:38pm UTC](https://discuss.elastic.co/t/getting-custom-filebeat-module-into-production/275740/4 "2021-07-15T12:38:30Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed. ---
**Author:** ![Suganthi\_Krishnavath](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/suganthi_krishnavath/32/91965_2.png) [@Suganthi\_Krishnavath](https://discuss.elastic.co/u/Suganthi_Krishnavath)\ **Post date:** [July 22, 2021, 12:51am UTC](https://discuss.elastic.co/t/getting-custom-filebeat-module-into-production/275740/5 "2021-07-22T00:51:31Z")
---
**Author:** ![Mario\_Castro](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mario_castro/32/35107_2.png) [@Mario\_Castro](https://discuss.elastic.co/u/Mario_Castro)\ **Post date:** [July 22, 2021, 9:02am UTC](https://discuss.elastic.co/t/getting-custom-filebeat-module-into-production/275740/6 "2021-07-22T09:02:36Z")
Hi @kelvins sorry for the late reply. Why don´t you open a PR to contribute your module to the community? It's extremely difficult to help if we cannot see your code and what might be wrong. Maybe you are lacking some `make update` command in the filebeat root. Now that you know that your module is loading, I also suggest to set console as output (remove elasticsearch as output) and see if your message appears in the console as expected: ```auto # filebeat.yml file output.console: pretty: true ``` You have to add the fields of your module in the `_meta/fields.yml` of your module folder too, then run the `make update` I mentioned before. ---
**Author:** ![kelvins](https://avatars.discourse-cdn.com/v4/letter/k/f19dbf/32.png) [@kelvins](https://discuss.elastic.co/u/kelvins)\ **Post date:** [July 28, 2021, 10:13am UTC](https://discuss.elastic.co/t/getting-custom-filebeat-module-into-production/275740/7 "2021-07-28T10:13:11Z")
Mario Thanks for getting back to me. I will santisie the test files, and then upload to GitHub for you to have a look at. Problem with the module at the moment, is some of the fields are ‘made up’ as cannot find a full spec for it. Will get back to you in a day or so. ![~WRD000.jpg](https://us1.discourse-cdn.com/elastic/original/3X/5/2/52d5f1c08649212fe6768d2cacdcf8df711810d9.jpeg) ![image001.png](https://us1.discourse-cdn.com/elastic/original/3X/0/a/0af250313c8687c5fb3250b0f722ccb4722a07e0.png) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 4, 2022, 8:31am UTC](https://discuss.elastic.co/t/getting-custom-filebeat-module-into-production/275740/8 "2022-11-04T08:31:05Z")