Dev environment - testsuite failing on filebeat

mariomechoulam · May 2, 2018, 2:39pm

Hello everyone!

I would love to contribute to Beats, but I am having a hard time setting up the dev environment and validating the tests before starting any work. This would be my first Go project, apologies if something is not clear or there are basic mistakes.
I followed the steps described here and ended up with the repo cloned in

/home/<user>/git/go/src/github.com/elastic/

with my PATH and GOPATH being

/usr/lib/go-1.10/bin:/home/<user>/git/go/bin/
/home/<user>/git/go

Details about the versions of the tools I run:
OS

Linux 4.10.0-37-generic #41~16.04.1-Ubuntu SMP Fri Oct 6 22:42:59 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Go

go version go1.10 linux/amd64

Python

Python 2.7.12

Pip

pip 10.0.1 from /home/<user>/.local/lib/python2.7/site-packages/pip (python 2.7)

Virtualenv

15.2.0

Docker

Version: 18.03.1-ce
API version: 1.37 (minimum version 1.12)
Go version: go1.9.5
Git commit: 9ee9f40

I can run the make command fine inside the elastic/beats/filebeat/ directory, but make testsuite fails with:

...
# gotestcover is needed to fetch coverage for multiple packages
go get github.com/elastic/beats/vendor/github.com/pierrre/gotestcover
# testify is needed for unit and integration tests
go get github.com/elastic/beats/vendor/github.com/stretchr/testify/assert
# libcompose is needed for integration tests
go get github.com/docker/libcompose
...
ok  	github.com/elastic/beats/filebeat/harvester/encoding	0.005s	coverage: 63.6% of statements
command [go test -cover -coverprofile /tmp/gotestcover-062121212 github.com/elastic/beats/filebeat/config]: exit status 1
--- FAIL: TestMergeConfigFiles (0.00s)
        Error Trace:    config_test.go:97
	Error:      	Not equal: 
	            	expected: 4
	            	actual  : 0
	Test:       	TestMergeConfigFiles
FAIL
...
test failed
../libbeat/scripts/Makefile:169: recipe for target 'unit-tests' failed
make[1]: *** [unit-tests] Error 1
make[1]: Leaving directory '/home/<user>/git/go/src/github.com/elastic/beats/filebeat'
../libbeat/scripts/Makefile:255: recipe for target 'testsuite' failed
make: *** [testsuite] Error 2

Running make test in the same folder or make in the main directory also fail. Can provide all the full traces if needed. What does work though is running go test - but perhaps that's not the proper way to test the project.
I also tried running the go get github.com/... commands that appeared on the logs, maybe hinting something, but nothing did change even after that.

I've been banging my head for the last hours around this. Am I even running this right?
If anyone can help me out or point me in the right direction that would be awesome

steffens · May 2, 2018, 4:05pm

Which OS are you using?

Running go test, executes the tests in the host environment. Running make test uses docker to start the different services and runs the tests in docker. The unit tests are written in go, but a many system tests are written in python.

It's one unit test in the github.com/elastic/beats/filebeat/config package failing. I did just run the tests unit test myself without any problems. Did you apply any modifications?

You can execute all unit tests via make unit-tests or locally via:

cd $GOPATH/src/github.com/elastic/beats/filebeat
go test ./...

Note the use of ./.... This means, test all packages recursively. Without any path/package name, go test will only run and test your current package/directory.

Complete log output can be helpful as well.

mariomechoulam · May 3, 2018, 7:40am

Hi Steffen and thanks for the quick reply!

I am running Linux Mint 18.2 (based on Ubuntu 16.04). And I just checkout the master branch and ran make (success) and make testsuite (failing) as suggested in the contribution guidelines; so no changes at all =/

The complete log output for make unit-tests
https://pastebin.com/8Tz9hXys

go test ./...

ok      github.com/elastic/beats/filebeat   (cached)
?       github.com/elastic/beats/filebeat/autodiscover  [no test files]
ok      github.com/elastic/beats/filebeat/autodiscover/builder/hints    (cached)
?       github.com/elastic/beats/filebeat/beater    [no test files]
?       github.com/elastic/beats/filebeat/channel   [no test files]
?       github.com/elastic/beats/filebeat/cmd   [no test files]
--- FAIL: TestMergeConfigFiles (0.00s)
        Error Trace:    config_test.go:97
    Error:          Not equal: 
                    expected: 4
                    actual  : 0
    Test:           TestMergeConfigFiles
FAIL
FAIL    github.com/elastic/beats/filebeat/config    0.004s
?       github.com/elastic/beats/filebeat/crawler   [no test files]
ok      github.com/elastic/beats/filebeat/fileset   (cached)
ok      github.com/elastic/beats/filebeat/harvester (cached)
ok      github.com/elastic/beats/filebeat/harvester/encoding    (cached)
ok      github.com/elastic/beats/filebeat/harvester/reader  (cached)
?       github.com/elastic/beats/filebeat/include   [no test files]
ok      github.com/elastic/beats/filebeat/input (cached)
?       github.com/elastic/beats/filebeat/input/docker  [no test files]
ok      github.com/elastic/beats/filebeat/input/file    (cached)
ok      github.com/elastic/beats/filebeat/input/log (cached)
?       github.com/elastic/beats/filebeat/input/redis   [no test files]
?       github.com/elastic/beats/filebeat/input/stdin   [no test files]
ok      github.com/elastic/beats/filebeat/input/syslog  (cached)
ok      github.com/elastic/beats/filebeat/input/tcp (cached)
?       github.com/elastic/beats/filebeat/input/udp [no test files]
?       github.com/elastic/beats/filebeat/inputsource   [no test files]
ok      github.com/elastic/beats/filebeat/inputsource/tcp   (cached)
ok      github.com/elastic/beats/filebeat/inputsource/udp   (cached)
ok      github.com/elastic/beats/filebeat/processor/add_kubernetes_metadata (cached)
?       github.com/elastic/beats/filebeat/prospector    [no test files]
?       github.com/elastic/beats/filebeat/registrar [no test files]
?       github.com/elastic/beats/filebeat/scripts/generator [no test files]
ok      github.com/elastic/beats/filebeat/scripts/generator/fields  (cached)
?       github.com/elastic/beats/filebeat/scripts/generator/fileset [no test files]
?       github.com/elastic/beats/filebeat/scripts/generator/module  [no test files]
ok      github.com/elastic/beats/filebeat/scripts/tester    (cached)
ok      github.com/elastic/beats/filebeat/util  (cached)

Is there a way I can see a more verbose output?

Thanks for your time!

steffens · May 3, 2018, 11:51am

For more verbose output you can run go test -v.

I have no idea why this particular unit test is failing for you. This one exists for quite a long time and the functionality it's testing wasn't touched for like a year or longer.

What happens if you run go test -cover github.com/elastic/beats/filebeat/config (can be run from any directory, as the command uses the complete package name).

The make unit-tests output doesn't add anything new here.

The test failing is TestMergeConfigFiles in filebeat/config/config_test.go. Maybe you can use fmt.Println in order to print intermediate values. Also print the absPath and check if the directory and test files actually exist.

When checking out from master, did you modify or delete and files?

mariomechoulam · May 3, 2018, 12:57pm

Once I checked out master I did not modify the contents or delete any file.

However, the first few times I ran the tests I saw some failures with a recommended "action" in the log itself. I think there is already a topic or an issue opened somewhere relating to the umask.
Basically I ran a few commands like the one below and those tests stopped failing anymore:

chmod go-w /home/<user>/git/go/src/github.com/elastic/beats/filebeat/module/auditd/log/manifest.yml

I sincerely hope that this does not count as a modification and that I did not make things worse.

So, running the above commands I get the following output:

go test ./... -v
go_test_minus_v - Pastebin.com

go test -cover github.com/elastic/beats/filebeat/config

> --- FAIL: TestMergeConfigFiles (0.00s)
>     Error Trace:    config_test.go:97
> 	Error:      	Not equal: 
> 	            	expected: 4
> 	            	actual  : 0
> 	Test:       	TestMergeConfigFiles
> FAIL
> coverage: 55.8% of statements
> FAIL	github.com/elastic/beats/filebeat/config	0.003s

I'll keep working on the logs and your other suggestions meanwhile. The file is there, so I'll play around with it.

ruflin · May 3, 2018, 1:24pm

Could you modify the test as following:

	config := &Config{}
	err = mergeConfigFiles(files, config)
	assert.NoError(t, err)

	assert.Equal(t, 4, len(config.Inputs))

We didn't check the error in the test. But it seem as in your case the merging fails, there could be an error that gives us more details on what is happening here.

mariomechoulam · May 3, 2018, 1:38pm

As soon as I added the extra logs it was clear

--- FAIL: TestMergeConfigFiles (0.00s)
    Error Trace:    config_test.go:96
Error:      	Received unexpected error:
            	Failed to read /home/<user>/git/go/src/github.com/elastic/beats/filebeat/tests/files/config.yml: config file ("/home/<user>/git/go/src/github.com/elastic/beats/filebeat/tests/files/config.yml") can only be writable by the owner but the permissions are "-rw-rw-r--" (to fix the permissions use: 'chmod go-w /home/<user>/git/go/src/github.com/elastic/beats/filebeat/tests/files/config.yml')
Test:       	TestMergeConfigFiles
    Error Trace:    config_test.go:98
Error:      	Not equal: 
            	expected: 4
            	actual  : 0
Test:       	TestMergeConfigFiles

After running the suggested command all the filebeat tests are now passing.
Probably the rest of the failures I am seeing in other beats share the same cause. Will update here after I run through them all.

Thank you both for the support!

ruflin · May 3, 2018, 2:05pm

Nice. Could you open a PR with these test changes? Will make it easier for someone getting into this into the future.

mariomechoulam · May 3, 2018, 2:16pm

For the sake of completeness and for anyone landing on this topic and with the same problems I am facing, here are the details of the remaining failures I was seeing, as well as the solutions that worked for me.

In addition to filebeat, I had failing test in auditbeat and packetbeat:

--- FAIL: TestEventReader (1.00s)
--- FAIL: TestEventReader/created (1.00s)
	eventreader_test.go:224: timed-out waiting for event
		github.com/elastic/beats/auditbeat/module/file_integrity.readTimeout
			/home/<user>/git/go/src/github.com/elastic/beats/auditbeat/module/file_integrity/eventreader_test.go:224
		github.com/elastic/beats/auditbeat/module/file_integrity.TestEventReader.func1
			/home/<user>/git/go/src/github.com/elastic/beats/auditbeat/module/file_integrity/eventreader_test.go:58
		testing.tRunner
			/usr/lib/go-1.10/src/testing/testing.go:777
		runtime.goexit
			/usr/lib/go-1.10/src/runtime/asm_amd64.s:2361
FAIL
FAIL	github.com/elastic/beats/auditbeat/module/file_integrity	11.052s
--- FAIL: TestNonRecursive (0.00s)
	monitor_test.go:343: no space left on device
--- FAIL: TestRecursive (0.00s)
	monitor_test.go:343: 1 error: failed adding watcher to '/tmp/monitor750016896': no space left on device
--- FAIL: TestRecursiveNoFollowSymlink (0.00s)
	monitor_test.go:343: 1 error: failed adding watcher to '/tmp/monitor117824223': no space left on device
--- FAIL: TestRecursiveSubdirPermissions (0.00s)
	monitor_test.go:343: 1 error: failed adding watcher to '/tmp/monitor512073321': no space left on device
FAIL
FAIL	github.com/elastic/beats/auditbeat/module/file_integrity/monitor	0.011s
...
FAIL	github.com/elastic/beats/packetbeat [build failed]
...
FAIL	github.com/elastic/beats/packetbeat/sniffer [build failed]

The solution for packetbeat was installing the missing libpcap-dev dependency in my OS as pcap.h was nowhere to be found and thus no packets could be captured.

For auditbeat, apparently I was hitting up the limit of folder watches in my distro (perhaps running too much stuff). To temporarily increase them I used:

sysctl fs.inotify.max_user_watches=524288

Although in order to make it permanent you would probably need to edit /etc/sysctl.conf. I had plenty of space in the hard disk so that was definitely not an issue.

That's it, all the tests passing now

Thanks again for all the time and support!

EDIT: @ruflin yes, of course

ruflin · May 4, 2018, 7:50am

Thanks so much for posting this here and sharing it with the community

system · June 1, 2018, 7:50am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.