Create new Filebeat

[saffatechy]

Hi @jsoriano

Thanks for your support so far.

I'm still getting an error, new error is this now:

        for key in flat.keys():
            metaKey = key.startswith('@metadata.')
            # Range keys as used in 'date_range' etc will not have docs of course
            isRangeKey = key.split('.')[-1] in ['gte', 'gt', 'lte', 'lt']
            if not(is_documented(key, expected_fields) or metaKey or isRangeKey):
>               raise Exception("Key '{}' found in event is not documented!".format(key))
E               Exception: Key 'bizswitch.thread' found in event is not documented!

../libbeat/tests/system/beat/beat.py:729: Exception
------------------------------------------------------------------------------------------- Captured stdout call -------------------------------------------------------------------------------------------
Using elasticsearch: http://localhost:9200
Testing bizswitch/log on /home/saffatechy/projects/go/src/github.com/elastic/beats/filebeat/tests/system/../../module/bizswitch/log/test/BizSwitch.log
========================================================================================= short test summary info ==========================================================================================
FAILED tests/system/test_modules.py::Test::test_fileset_file_0_bizswitch - Exception: Key 'bizswitch.thread' found in event is not documented!
====================================================================================== 1 failed in 165.39s (0:02:45) =======================================================================================

Even though the key can be found in the fields.yml file.

Any ideas?

[jsoriano (Jaime Soriano)]

Try to run mage fields, and recompile filebeat test binary with go test -c before running the test.

[saffatechy]

Hi @jsoriano

I did do that.

i.e:

make create-fields MODULE=bizswitch FILESET=log

Is that not correct?

[saffatechy]

Then after that:

make update

[jsoriano (Jaime Soriano)]

Yep, these commands should be enough, but still try explicitly running mage fields and go test -c before running the test.

[saffatechy]

@jsoriano

go test -c

Hi Jaime

Unfortunately still the same result:

        for key in flat.keys():
            metaKey = key.startswith('@metadata.')
            # Range keys as used in 'date_range' etc will not have docs of course
            isRangeKey = key.split('.')[-1] in ['gte', 'gt', 'lte', 'lt']
            if not(is_documented(key, expected_fields) or metaKey or isRangeKey):
>               raise Exception("Key '{}' found in event is not documented!".format(key))
E               Exception: Key 'bizswitch.thread' found in event is not documented!

../libbeat/tests/system/beat/beat.py:729: Exception
------------------------------------------------------------------------------------------- Captured stdout call -------------------------------------------------------------------------------------------
Using elasticsearch: http://localhost:9200
Testing bizswitch/log on /home/justint/projects/go/src/github.com/elastic/beats/filebeat/tests/system/../../module/bizswitch/log/test/BizSwitch.log
========================================================================================= short test summary info ==========================================================================================
FAILED tests/system/test_modules.py::Test::test_fileset_file_0_bizswitch - Exception: Key 'bizswitch.thread' found in event is not documented!
====================================================================================== 1 failed in 142.49s (0:02:22) =======================================================================================

Are you sure you haven't missed anything?

[jsoriano (Jaime Soriano)]

@saffatechy

Are you sure you haven't missed anything?

I have tried again with these steps:

• Check-out your code in GitHub - saffatechy/beats
• Apply the patch I suggested in this comment.
• In the filebeat directory: mage fields
• Also in the filebeat directory: go test -c

After this, the tests work for me.

If I modify the fields.yml file to remove the bizswitch.thread field, I obtain the same error as you do. What seems to indicate that your fields are not updated.

Have you made some other change that is not pushed in GitHub - saffatechy/beats?

[saffatechy]

Hi @jsoriano

My sincere apologies for the massively delayed reply.
Please find my latest pushed master branch:

saffatechy/beats

[saffatechy]

bump @jsoriano

[jsoriano (Jaime Soriano)]

Hey @saffatechy,

In your last change, you modified the pipeline as I suggested, but you introduced some changes in the fields.yml file too. At least the thread and category fields need to be defined line this, without the bizswitch preffix that is automatically added:

- name: thread
  description: BizSwitch Thread
  example: Timer-21
  type: keyword
- name: category
  description: Please add description
  example: Please add example
  type: keyword

[saffatechy]

HI @jsoriano

I modified it because when running:
make create-fields MODULE=bizswitch FILESET=log

For some reason, from the pipeline it is not creating the "thread" field in the fields.yml file.

So upon deleting the _meta/fields.yml file and rerunning said command, I am in the same situation.

I do see your point on the prefix that is added automatically after going through the above.

[saffatechy]

Hi @jsoriano

Sorry to pester, any advice?

[jsoriano (Jaime Soriano)]

Hey @saffatechy,

Sorry, but I don't know what else to advice

If I download your code, and apply the changes on the fields.yml file of the bizswitch module, everything works for me. No need to run make create-fields again or anything.

[saffatechy]

@jsoriano

your

Hi @jsoriano

Any possibility you could push your change into my master branch?

[jsoriano (Jaime Soriano)]

@saffatechy

Any possibility you could push your change into my master branch?

I don't have permissions for that. But really, the change is only to modify filebeat/module/bizswitch/log/_meta/fields.yml with this change:

diff --git a/filebeat/module/bizswitch/log/_meta/fields.yml b/filebeat/module/bizswitch/log/_meta/fields.yml
index 740bb7b8f0..f15574903a 100644
--- a/filebeat/module/bizswitch/log/_meta/fields.yml
+++ b/filebeat/module/bizswitch/log/_meta/fields.yml
@@ -5,11 +5,11 @@
   description: Please add description
   example: Please add example
   type: keyword
-- name: bizswitch.thread
+- name: thread
   description: BizSwitch Thread
   example: Timer-21
   type: keyword
-- name: bizswitch.category
+- name: category
   description: Please add description
   example: Please add example
   type: keyword

(Replace bizswitch.thread with thread and bizswitch.category with category).

Then run mage fields so some files are updated, and that's it. Tests can be run and I see events like this one:

    {
        "@timestamp": "2020-08-20T00:00:00.010-02:00",
        "bizswitch.category": "PropertySettingJobFactory",
        "bizswitch.thread": "BizSwitchQuartzScheduler_QuartzSchedulerThread",
        "event.dataset": "bizswitch.log",
        "event.module": "bizswitch",
        "event.timezone": "-02:00",
        "fileset.name": "log",
        "input.type": "log",
        "log.level": "DEBUG",
        "log.offset": 0,
        "message": "Producing instance of Job 'IpayXml_IndraJsonTranslator.IpayXml_IndraJsonTranslator.BalanceCheck', class=com.innoforge.bizswitch.components.translators.ipayxml_indrajsontranslator.IpayXml_IndraJsonTranslator$BalanceCheck",
        "service.type": "bizswitch"
    }