"make package" broken for community beats?

[fjgal]

Hi,

I'm trying to build a community beat based on metribeat and it fails. To reproduce the issue I created a dummy beat following these steps:

$ cd $GOPATH/src/github.com/someuser
$ cookiecutter $GOPATH/src/github.com/elastic/beats/generate/metricbeat/metricset
project_name [Examplebeat]: somebeat
github_name [your-github-name]: someuser
beat [somebeat]: somebeat
beat_path [github.com/someuser]: github.com/someuser
full_name [Firstname Lastname]: Some User

$ cd somebeat/
$ make setup
$ make update
$ make package

I found two issues:

1) incorrect destination directory in build.sh

"make package" fails as follows:

Working directory=/go/src/github.com
Copying main source folder /source to folder /go/src/github.com
Fetching dependencies...
Execute /scripts/before_build.sh github.com/someuser
/scripts/before_build.sh: 13: cd: can't cd to /go/src/github.com/someuser
make[1]: *** [prepare-package-cgo] Error 2
make: *** [package] Error 2

The issue seems to be that the argument passed to build.sh is the value of the BEAT_DIR variable in the generated Makefile that doesn't contain the beat name.

Extract from generated Makefile

BEATNAME=somebeat
BEAT_DIR=github.com/someuser

build.sh is extracting the directory name using the dirname command, thus removing the last part of the BEAT_DIR variable

Extract from build.sh

BEAT_PATH=$1
DST_FOLDER=`dirname $GOPATH/src/$BEAT_PATH`
GIT_REPO=$BEAT_PATH

In the example BEAT_PATH is "github.com/someuser", DST_FOLDER will be "/go/src/github.com". build.sh creates the DST_FOLDER (/go/src/github.com). Then before_build.sh is called with $BEAT_PATH (github.com/someuser) as argument and tries to change directory to /go/src/${1} (/go/src/github.com/someuser) and fails.

To workaround the issue I modified build.sh to:

BEAT_NAME=$PACK
BEAT_PATH=$1/$BEAT_NAME
DST_FOLDER=`dirname $GOPATH/src/$BEAT_PATH`
GIT_REPO=$BEAT_PATH

There are multiple ways in which this could be solved, I tried a few, but this is the only one I found that didn't have side-effects. Not sure if this is the correct way though.

After applying the above workaround, I faced a second issue.

2) make package tries to modify metricbeat specific files (metricbeat-win.yml and metricbeat-win.full.yml)

It fails with the following error:

[...]
Execute /scripts/before_build.sh github.com/someuser/somebeat
install -d -m 755 /build/homedir/scripts/
install -m 755 ./vendor/github.com/elastic/beats/libbeat/scripts/migrate_beat_config_1_x_to_5_0.py /build/homedir/scripts/
Compiling import_dashboards for windows/amd64
Compiling import_dashboards for windows/386
Compiling import_dashboards for darwin/amd64
Cloning into '/go/src/github.com/tsg/gotpl'...
# disable the system/load metricset on windows
sed -i.bk 's/- load/#- load/' /build/metricbeat-win.yml
sed: can't read /build/metricbeat-win.yml: No such file or directory
make: *** [before-build] Error 2
make[1]: *** [prepare-package-cgo] Error 2
make: *** [package] Error 2

I tracked the problem down to metricbeat/Makefile (line 56-63):

# This is called by the beats packer before building starts
.PHONY: before-build
before-build:
	# disable the system/load metricset on windows
	sed -i.bk 's/- load/#- load/' $(PREFIX)/metricbeat-win.yml
	rm $(PREFIX)/metricbeat-win.yml.bk
	sed -i.bk 's/- load/#- load/' $(PREFIX)/metricbeat-win.full.yml
	rm $(PREFIX)/metricbeat-win.full.yml.bk

metricbeat/Makefile is included in the generated Makefile for the community beat, and when called it fails since the *metricbeat-win*.yml don't exist for the community beat:

# disable the system/load metricset on windows
    sed -i.bk 's/- load/#- load/' /build/metricbeat-win.yml
    sed: can't read /build/metricbeat-win.yml: No such file or directory

The sed commands should only be executed when packaging metricbeat and not for other beats based on it. A possible solution (probably not the cleanest) is to introduce a condition on the BEATNAME, changing the above lines to:

# This is called by the beats packer before building starts
.PHONY: before-build
before-build:
ifeq ($(BEATNAME), metricbeat)
        # disable the system/load metricset on windows
        sed -i.bk 's/- load/#- load/' $(PREFIX)/metricbeat-win.yml
        rm $(PREFIX)/metricbeat-win.yml.bk
        sed -i.bk 's/- load/#- load/' $(PREFIX)/metricbeat-win.full.yml
        rm $(PREFIX)/metricbeat-win.full.yml.bk
endif

This issue seems to be introduced by dd43dd1

I tested with the two changes and the community beat gets properly packaged.

If you confirm these are valid issues I can submit separate github issues and pull requests for each one.

Thank you!

/F

[ruflin]

Thanks for testing this and also finding some workaround. These are definitively valid issue. It would be great if you could open a PR / Github issue for it.

Especially the BEAT_PATH part is tricky (and ugly). We already have some tweaks inside to support community beats, but we missed the ones based on metricbeat Lets discuss / test these changes on a PR, then we see also if it does not break our packaging

[fjgal]

Thanks @ruflin for the feedback. My pleasure to help improving this already great product.

I created Github issues and PRs:

Issue#1

Issue: Wrong BEAT_PATH for metricbeat based community beats

Fixed BEAT_PATH issue when packaging metricbeat based community beats

Issue#2

Issue: "make package" broken for metricbeat based community beats

Fixed "make package" for metribeat based community beats

[ruflin]

Thanks. I continued the discussions there.