Testing beat module test_modules.Test freezes

[Jim_Ivey]

I'm trying to develop a module for filebeat. Running make testsuite, the furthest it ever gets is testing my module, where it just freezes -- for hours. For example:

test_fileset_file_071_postgresql (test_modules.Test) ... ok (2.8172s)
test_fileset_file_072_postgresql (test_modules.Test) ... ok (2.8613s)
test_fileset_file_073_postgresql (test_modules.Test) ... ok (2.5822s)
test_fileset_file_074_icinga (test_modules.Test) ... ok (2.7147s)
test_fileset_file_075_icinga (test_modules.Test) ... ok (2.5620s)
test_fileset_file_076_icinga (test_modules.Test) ... ok (2.6743s)
test_fileset_file_077_my_module (test_modules.Test) ... 

I'm guessing there's something fundamental that's wrong with my module, but I can only guess without more information. Where can I find more information about what's happening in my module?

I set up my dev environment as described here within the last couple weeks. So my version of everything ought to be up to date.

Any help is greatly appreciated.

[Mario_Castro]

Hi @Jim_Ivey

The kind of freezing errors in Filebeat are usually syntax errors. For example, an incorrect Grok pattern (a simple comma can make it fail already). You can test the patterns here https://grokconstructor.appspot.com/do/match for example (be aware because that webpage uses Ruby Grok which is slightly different to the one on Filebeat but still 95% compatible).

Another freezing issue can be in the input example log file like this example. They must end with a newline.

I encourage you to open a PR in the repo if you think it can be contributed, we are happy to assist people in developing their modules to the repo. It is easier to help too

By the way, you don't need to run the entire testsuite every time you want to check your code. For example, you can run nginx with the following oneliner in filebeat folder:
TESTING_FILEBEAT_MODULES=nginx INTEGRATION_TESTS=1 nosetests -v -d -s tests/system/test_modules.py

[Jim_Ivey]

Thanks! It turns out I was using the timestamp processor described here, and whatever I'm using hadn't heard of it.

Your last tip really speeds thing up for me! Thanks!

FWIW, playing with find and grep, I found a lot of information in build/system-tests/run/test_modules.Test.test_fileset_file_077_my_module/output.log Though, after running the single-module test, that file is empty.

Duh, my bad: since I'm just testing a single module, it's no longer at ...file_077...; it's now at ...file_0.... Thanks again!

[Jim_Ivey]

The single-module testing is failing due to a missing file: ls: cannot access '.../go/src/github.com/elastic/beats/filebeat/filebeat.test': No such file or directory

I've run make, make update, and make fmt and still no file. I've even run make testsuite just in case that generates the file -- no luck.

What am I missing? Any help is greatly appreciated.