# Create new Filebeat **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [August 24, 2020, 9:19am UTC](https://discuss.elastic.co/t/create-new-filebeat/246081 "2020-08-24T09:19:28Z")\ **Posts on this page:** 16\ **Page:** 2
**Author:** ![saffatechy](https://avatars.discourse-cdn.com/v4/letter/s/58956e/32.png) [@saffatechy](https://discuss.elastic.co/u/saffatechy)\ **Post date:** [September 22, 2020, 8:56am UTC](https://discuss.elastic.co/t/create-new-filebeat/246081/21 "2020-09-22T08:56:16Z")
Hi @jsoriano Thanks for your support so far. I'm still getting an error, new error is this now: ```auto for key in flat.keys(): metaKey = key.startswith('@metadata.') # Range keys as used in 'date_range' etc will not have docs of course isRangeKey = key.split('.')[-1] in ['gte', 'gt', 'lte', 'lt'] if not(is_documented(key, expected_fields) or metaKey or isRangeKey): > raise Exception("Key '{}' found in event is not documented!".format(key)) E Exception: Key 'bizswitch.thread' found in event is not documented! ../libbeat/tests/system/beat/beat.py:729: Exception ------------------------------------------------------------------------------------------- Captured stdout call ------------------------------------------------------------------------------------------- Using elasticsearch: http://localhost:9200 Testing bizswitch/log on /home/saffatechy/projects/go/src/github.com/elastic/beats/filebeat/tests/system/../../module/bizswitch/log/test/BizSwitch.log ========================================================================================= short test summary info ========================================================================================== FAILED tests/system/test_modules.py::Test::test_fileset_file_0_bizswitch - Exception: Key 'bizswitch.thread' found in event is not documented! ====================================================================================== 1 failed in 165.39s (0:02:45) ======================================================================================= ``` Even though the key can be found in the fields.yml file. Any ideas? ---
**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\ **Post date:** [September 22, 2020, 9:27am UTC](https://discuss.elastic.co/t/create-new-filebeat/246081/22 "2020-09-22T09:27:02Z")
Try to run `mage fields`, and recompile filebeat test binary with `go test -c` before running the test. ---
**Author:** ![saffatechy](https://avatars.discourse-cdn.com/v4/letter/s/58956e/32.png) [@saffatechy](https://discuss.elastic.co/u/saffatechy)\ **Post date:** [September 22, 2020, 10:02am UTC](https://discuss.elastic.co/t/create-new-filebeat/246081/23 "2020-09-22T10:02:34Z")
Hi @jsoriano I did do that. i.e: make create-fields MODULE=bizswitch FILESET=log Is that not correct? ---
**Author:** ![saffatechy](https://avatars.discourse-cdn.com/v4/letter/s/58956e/32.png) [@saffatechy](https://discuss.elastic.co/u/saffatechy)\ **Post date:** [September 22, 2020, 10:06am UTC](https://discuss.elastic.co/t/create-new-filebeat/246081/24 "2020-09-22T10:06:57Z")
Then after that: make update ---
**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\ **Post date:** [September 28, 2020, 11:44am UTC](https://discuss.elastic.co/t/create-new-filebeat/246081/25 "2020-09-28T11:44:11Z")
Yep, these commands should be enough, but still try explicitly running `mage fields` and `go test -c` before running the test. ---
**Author:** ![saffatechy](https://avatars.discourse-cdn.com/v4/letter/s/58956e/32.png) [@saffatechy](https://discuss.elastic.co/u/saffatechy)\ **Post date:** [October 1, 2020, 9:43am UTC](https://discuss.elastic.co/t/create-new-filebeat/246081/26 "2020-10-01T09:43:32Z")
> [@jsoriano](#): > > `go test -c` Hi Jaime Unfortunately still the same result: ```auto for key in flat.keys(): metaKey = key.startswith('@metadata.') # Range keys as used in 'date_range' etc will not have docs of course isRangeKey = key.split('.')[-1] in ['gte', 'gt', 'lte', 'lt'] if not(is_documented(key, expected_fields) or metaKey or isRangeKey): > raise Exception("Key '{}' found in event is not documented!".format(key)) E Exception: Key 'bizswitch.thread' found in event is not documented! ../libbeat/tests/system/beat/beat.py:729: Exception ------------------------------------------------------------------------------------------- Captured stdout call ------------------------------------------------------------------------------------------- Using elasticsearch: http://localhost:9200 Testing bizswitch/log on /home/justint/projects/go/src/github.com/elastic/beats/filebeat/tests/system/../../module/bizswitch/log/test/BizSwitch.log ========================================================================================= short test summary info ========================================================================================== FAILED tests/system/test_modules.py::Test::test_fileset_file_0_bizswitch - Exception: Key 'bizswitch.thread' found in event is not documented! ====================================================================================== 1 failed in 142.49s (0:02:22) ======================================================================================= ``` Are you sure you haven't missed anything? ---
**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\ **Post date:** [October 1, 2020, 11:32am UTC](https://discuss.elastic.co/t/create-new-filebeat/246081/27 "2020-10-01T11:32:41Z")
> [@saffatechy](#): > > Are you sure you haven't missed anything? I have tried again with these steps: - Check-out your code in [GitHub - saffatechy/beats](https://github.com/saffatechy/beats) - Apply the patch I suggested in [this comment](https://discuss.elastic.co/t/create-new-filebeat/246081/18). - In the filebeat directory: `mage fields` - Also in the filebeat directory: `go test -c` After this, the tests work for me. If I modify the `fields.yml` file to remove the `bizswitch.thread` field, I obtain the same error as you do. What seems to indicate that your fields are not updated. Have you made some other change that is not pushed in [GitHub - saffatechy/beats](https://github.com/saffatechy/beats)? ---
**Author:** ![saffatechy](https://avatars.discourse-cdn.com/v4/letter/s/58956e/32.png) [@saffatechy](https://discuss.elastic.co/u/saffatechy)\ **Post date:** [October 14, 2020, 10:34am UTC](https://discuss.elastic.co/t/create-new-filebeat/246081/28 "2020-10-14T10:34:40Z")
Hi @jsoriano My sincere apologies for the massively delayed reply. Please find my latest pushed master branch: > **[saffatechy/beats](https://github.com/saffatechy/beats)** > > Contribute to saffatechy/beats development by creating an account on GitHub. ---
**Author:** ![saffatechy](https://avatars.discourse-cdn.com/v4/letter/s/58956e/32.png) [@saffatechy](https://discuss.elastic.co/u/saffatechy)\ **Post date:** [October 19, 2020, 9:38am UTC](https://discuss.elastic.co/t/create-new-filebeat/246081/29 "2020-10-19T09:38:28Z")
bump @jsoriano ---
**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\ **Post date:** [October 19, 2020, 4:28pm UTC](https://discuss.elastic.co/t/create-new-filebeat/246081/30 "2020-10-19T16:28:04Z")
Hey @saffatechy, In your last change, you modified the pipeline as I suggested, but you introduced some changes in the `fields.yml` file too. At least the `thread` and `category` fields need to be defined line this, without the `bizswitch` preffix that is automatically added: ```auto - name: thread description: BizSwitch Thread example: Timer-21 type: keyword - name: category description: Please add description example: Please add example type: keyword ``` ---
**Author:** ![saffatechy](https://avatars.discourse-cdn.com/v4/letter/s/58956e/32.png) [@saffatechy](https://discuss.elastic.co/u/saffatechy)\ **Post date:** [October 19, 2020, 5:47pm UTC](https://discuss.elastic.co/t/create-new-filebeat/246081/31 "2020-10-19T17:47:30Z")
HI @jsoriano I modified it because when running: make create-fields MODULE=bizswitch FILESET=log For some reason, from the pipeline it is not creating the "thread" field in the fields.yml file. So upon deleting the \_meta/fields.yml file and rerunning said command, I am in the same situation. I do see your point on the prefix that is added automatically after going through the above. ---
**Author:** ![saffatechy](https://avatars.discourse-cdn.com/v4/letter/s/58956e/32.png) [@saffatechy](https://discuss.elastic.co/u/saffatechy)\ **Post date:** [October 21, 2020, 2:12pm UTC](https://discuss.elastic.co/t/create-new-filebeat/246081/32 "2020-10-21T14:12:11Z")
Hi @jsoriano Sorry to pester, any advice? ---
**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\ **Post date:** [October 21, 2020, 5:01pm UTC](https://discuss.elastic.co/t/create-new-filebeat/246081/33 "2020-10-21T17:01:02Z")
Hey @saffatechy, Sorry, but I don't know what else to advice ☹ If I download your code, and apply the changes on the `fields.yml` file of the `bizswitch` module, everything works for me. No need to run `make create-fields` again or anything. ---
**Author:** ![saffatechy](https://avatars.discourse-cdn.com/v4/letter/s/58956e/32.png) [@saffatechy](https://discuss.elastic.co/u/saffatechy)\ **Post date:** [October 21, 2020, 8:17pm UTC](https://discuss.elastic.co/t/create-new-filebeat/246081/34 "2020-10-21T20:17:49Z")
> [@jsoriano](#): > > your Hi @jsoriano Any possibility you could push your change into my master branch? ---
**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\ **Post date:** [October 22, 2020, 8:01am UTC](https://discuss.elastic.co/t/create-new-filebeat/246081/35 "2020-10-22T08:01:22Z")
> [@saffatechy](#): > > Any possibility you could push your change into my master branch? I don't have permissions for that. But really, the change is only to modify `filebeat/module/bizswitch/log/_meta/fields.yml` with this change: ```auto diff --git a/filebeat/module/bizswitch/log/_meta/fields.yml b/filebeat/module/bizswitch/log/_meta/fields.yml index 740bb7b8f0..f15574903a 100644 --- a/filebeat/module/bizswitch/log/_meta/fields.yml +++ b/filebeat/module/bizswitch/log/_meta/fields.yml @@ -5,11 +5,11 @@ description: Please add description example: Please add example type: keyword -- name: bizswitch.thread +- name: thread description: BizSwitch Thread example: Timer-21 type: keyword -- name: bizswitch.category +- name: category description: Please add description example: Please add example type: keyword ``` (Replace `bizswitch.thread` with `thread` and `bizswitch.category` with `category`). Then run `mage fields` so some files are updated, and that's it. Tests can be run and I see events like this one: ```auto { "@timestamp": "2020-08-20T00:00:00.010-02:00", "bizswitch.category": "PropertySettingJobFactory", "bizswitch.thread": "BizSwitchQuartzScheduler_QuartzSchedulerThread", "event.dataset": "bizswitch.log", "event.module": "bizswitch", "event.timezone": "-02:00", "fileset.name": "log", "input.type": "log", "log.level": "DEBUG", "log.offset": 0, "message": "Producing instance of Job 'IpayXml_IndraJsonTranslator.IpayXml_IndraJsonTranslator.BalanceCheck', class=com.innoforge.bizswitch.components.translators.ipayxml_indrajsontranslator.IpayXml_IndraJsonTranslator$BalanceCheck", "service.type": "bizswitch" } ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 19, 2020, 10:01am UTC](https://discuss.elastic.co/t/create-new-filebeat/246081/36 "2020-11-19T10:01:29Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed. [Previous page](https://discuss.elastic.co/t/create-new-filebeat/246081.md?page=1)