Create new Filebeat

saffatechy · September 22, 2020, 8:56am

Thanks for your support so far.

I'm still getting an error, new error is this now:

        for key in flat.keys():
            metaKey = key.startswith('@metadata.')
            # Range keys as used in 'date_range' etc will not have docs of course
            isRangeKey = key.split('.')[-1] in ['gte', 'gt', 'lte', 'lt']
            if not(is_documented(key, expected_fields) or metaKey or isRangeKey):
>               raise Exception("Key '{}' found in event is not documented!".format(key))
E               Exception: Key 'bizswitch.thread' found in event is not documented!

../libbeat/tests/system/beat/beat.py:729: Exception
------------------------------------------------------------------------------------------- Captured stdout call -------------------------------------------------------------------------------------------
Using elasticsearch: http://localhost:9200
Testing bizswitch/log on /home/saffatechy/projects/go/src/github.com/elastic/beats/filebeat/tests/system/../../module/bizswitch/log/test/BizSwitch.log
========================================================================================= short test summary info ==========================================================================================
FAILED tests/system/test_modules.py::Test::test_fileset_file_0_bizswitch - Exception: Key 'bizswitch.thread' found in event is not documented!
====================================================================================== 1 failed in 165.39s (0:02:45) =======================================================================================

Even though the key can be found in the fields.yml file.

Any ideas?

jsoriano · September 22, 2020, 9:27am

Try to run mage fields, and recompile filebeat test binary with go test -c before running the test.

saffatechy · September 22, 2020, 10:02am

Hi @jsoriano

I did do that.

i.e:

make create-fields MODULE=bizswitch FILESET=log

Is that not correct?

saffatechy · September 22, 2020, 10:06am

Then after that:

make update

jsoriano · September 28, 2020, 11:44am

Yep, these commands should be enough, but still try explicitly running mage fields and go test -c before running the test.

saffatechy · October 1, 2020, 9:43am

Hi Jaime

Unfortunately still the same result:

        for key in flat.keys():
            metaKey = key.startswith('@metadata.')
            # Range keys as used in 'date_range' etc will not have docs of course
            isRangeKey = key.split('.')[-1] in ['gte', 'gt', 'lte', 'lt']
            if not(is_documented(key, expected_fields) or metaKey or isRangeKey):
>               raise Exception("Key '{}' found in event is not documented!".format(key))
E               Exception: Key 'bizswitch.thread' found in event is not documented!

../libbeat/tests/system/beat/beat.py:729: Exception
------------------------------------------------------------------------------------------- Captured stdout call -------------------------------------------------------------------------------------------
Using elasticsearch: http://localhost:9200
Testing bizswitch/log on /home/justint/projects/go/src/github.com/elastic/beats/filebeat/tests/system/../../module/bizswitch/log/test/BizSwitch.log
========================================================================================= short test summary info ==========================================================================================
FAILED tests/system/test_modules.py::Test::test_fileset_file_0_bizswitch - Exception: Key 'bizswitch.thread' found in event is not documented!
====================================================================================== 1 failed in 142.49s (0:02:22) =======================================================================================

Are you sure you haven't missed anything?

jsoriano · October 1, 2020, 11:32am

I have tried again with these steps:

Check-out your code in GitHub - saffatechy/beats
Apply the patch I suggested in this comment.
In the filebeat directory: mage fields
Also in the filebeat directory: go test -c

After this, the tests work for me.

If I modify the fields.yml file to remove the bizswitch.thread field, I obtain the same error as you do. What seems to indicate that your fields are not updated.

Have you made some other change that is not pushed in GitHub - saffatechy/beats?

saffatechy · October 14, 2020, 10:34am

Hi @jsoriano

My sincere apologies for the massively delayed reply.
Please find my latest pushed master branch:

saffatechy · October 19, 2020, 9:38am

bump @jsoriano

jsoriano · October 19, 2020, 4:28pm

Hey @saffatechy,

In your last change, you modified the pipeline as I suggested, but you introduced some changes in the fields.yml file too. At least the thread and category fields need to be defined line this, without the bizswitch preffix that is automatically added:

- name: thread
  description: BizSwitch Thread
  example: Timer-21
  type: keyword
- name: category
  description: Please add description
  example: Please add example
  type: keyword

saffatechy · October 19, 2020, 5:47pm

HI @jsoriano

I modified it because when running:
make create-fields MODULE=bizswitch FILESET=log

For some reason, from the pipeline it is not creating the "thread" field in the fields.yml file.

So upon deleting the _meta/fields.yml file and rerunning said command, I am in the same situation.

I do see your point on the prefix that is added automatically after going through the above.

saffatechy · October 21, 2020, 2:12pm

Hi @jsoriano

Sorry to pester, any advice?

jsoriano · October 21, 2020, 5:01pm

Hey @saffatechy,

Sorry, but I don't know what else to advice

If I download your code, and apply the changes on the fields.yml file of the bizswitch module, everything works for me. No need to run make create-fields again or anything.

saffatechy · October 21, 2020, 8:17pm

Hi @jsoriano

Any possibility you could push your change into my master branch?

jsoriano · October 22, 2020, 8:01am

I don't have permissions for that. But really, the change is only to modify filebeat/module/bizswitch/log/_meta/fields.yml with this change:

diff --git a/filebeat/module/bizswitch/log/_meta/fields.yml b/filebeat/module/bizswitch/log/_meta/fields.yml
index 740bb7b8f0..f15574903a 100644
--- a/filebeat/module/bizswitch/log/_meta/fields.yml
+++ b/filebeat/module/bizswitch/log/_meta/fields.yml
@@ -5,11 +5,11 @@
   description: Please add description
   example: Please add example
   type: keyword
-- name: bizswitch.thread
+- name: thread
   description: BizSwitch Thread
   example: Timer-21
   type: keyword
-- name: bizswitch.category
+- name: category
   description: Please add description
   example: Please add example
   type: keyword

(Replace bizswitch.thread with thread and bizswitch.category with category).

Then run mage fields so some files are updated, and that's it. Tests can be run and I see events like this one:

    {
        "@timestamp": "2020-08-20T00:00:00.010-02:00",
        "bizswitch.category": "PropertySettingJobFactory",
        "bizswitch.thread": "BizSwitchQuartzScheduler_QuartzSchedulerThread",
        "event.dataset": "bizswitch.log",
        "event.module": "bizswitch",
        "event.timezone": "-02:00",
        "fileset.name": "log",
        "input.type": "log",
        "log.level": "DEBUG",
        "log.offset": 0,
        "message": "Producing instance of Job 'IpayXml_IndraJsonTranslator.IpayXml_IndraJsonTranslator.BalanceCheck', class=com.innoforge.bizswitch.components.translators.ipayxml_indrajsontranslator.IpayXml_IndraJsonTranslator$BalanceCheck",
        "service.type": "bizswitch"
    }

system · November 19, 2020, 10:01am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.