Hi,
I'm trying to build a community beat based on metribeat and it fails. To reproduce the issue I created a dummy beat following these steps:
$ cd $GOPATH/src/github.com/someuser
$ cookiecutter $GOPATH/src/github.com/elastic/beats/generate/metricbeat/metricset
project_name [Examplebeat]: somebeat
github_name [your-github-name]: someuser
beat [somebeat]: somebeat
beat_path [github.com/someuser]: github.com/someuser
full_name [Firstname Lastname]: Some User
$ cd somebeat/
$ make setup
$ make update
$ make package
I found two issues:
1) incorrect destination directory in build.sh
"make package" fails as follows:
Working directory=/go/src/github.com
Copying main source folder /source to folder /go/src/github.com
Fetching dependencies...
Execute /scripts/before_build.sh github.com/someuser
/scripts/before_build.sh: 13: cd: can't cd to /go/src/github.com/someuser
make[1]: *** [prepare-package-cgo] Error 2
make: *** [package] Error 2
The issue seems to be that the argument passed to build.sh is the value of the BEAT_DIR variable in the generated Makefile that doesn't contain the beat name.
Extract from generated Makefile
BEATNAME=somebeat
BEAT_DIR=github.com/someuser
build.sh is extracting the directory name using the dirname command, thus removing the last part of the BEAT_DIR variable
Extract from build.sh
BEAT_PATH=$1
DST_FOLDER=`dirname $GOPATH/src/$BEAT_PATH`
GIT_REPO=$BEAT_PATH
In the example BEAT_PATH is "github.com/someuser", DST_FOLDER will be "/go/src/github.com". build.sh creates the DST_FOLDER (/go/src/github.com). Then before_build.sh is called with $BEAT_PATH (github.com/someuser) as argument and tries to change directory to /go/src/${1} (/go/src/github.com/someuser) and fails.
To workaround the issue I modified build.sh to:
BEAT_NAME=$PACK
BEAT_PATH=$1/$BEAT_NAME
DST_FOLDER=`dirname $GOPATH/src/$BEAT_PATH`
GIT_REPO=$BEAT_PATH
There are multiple ways in which this could be solved, I tried a few, but this is the only one I found that didn't have side-effects. Not sure if this is the correct way though.
After applying the above workaround, I faced a second issue.
2) make package tries to modify metricbeat specific files (metricbeat-win.yml and metricbeat-win.full.yml)
It fails with the following error:
[...]
Execute /scripts/before_build.sh github.com/someuser/somebeat
install -d -m 755 /build/homedir/scripts/
install -m 755 ./vendor/github.com/elastic/beats/libbeat/scripts/migrate_beat_config_1_x_to_5_0.py /build/homedir/scripts/
Compiling import_dashboards for windows/amd64
Compiling import_dashboards for windows/386
Compiling import_dashboards for darwin/amd64
Cloning into '/go/src/github.com/tsg/gotpl'...
# disable the system/load metricset on windows
sed -i.bk 's/- load/#- load/' /build/metricbeat-win.yml
sed: can't read /build/metricbeat-win.yml: No such file or directory
make: *** [before-build] Error 2
make[1]: *** [prepare-package-cgo] Error 2
make: *** [package] Error 2
I tracked the problem down to metricbeat/Makefile (line 56-63):
# This is called by the beats packer before building starts
.PHONY: before-build
before-build:
# disable the system/load metricset on windows
sed -i.bk 's/- load/#- load/' $(PREFIX)/metricbeat-win.yml
rm $(PREFIX)/metricbeat-win.yml.bk
sed -i.bk 's/- load/#- load/' $(PREFIX)/metricbeat-win.full.yml
rm $(PREFIX)/metricbeat-win.full.yml.bk
metricbeat/Makefile is included in the generated Makefile for the community beat, and when called it fails since the *metricbeat-win*.yml don't exist for the community beat:
# disable the system/load metricset on windows
sed -i.bk 's/- load/#- load/' /build/metricbeat-win.yml
sed: can't read /build/metricbeat-win.yml: No such file or directory
The sed commands should only be executed when packaging metricbeat and not for other beats based on it. A possible solution (probably not the cleanest) is to introduce a condition on the BEATNAME, changing the above lines to:
# This is called by the beats packer before building starts
.PHONY: before-build
before-build:
ifeq ($(BEATNAME), metricbeat)
# disable the system/load metricset on windows
sed -i.bk 's/- load/#- load/' $(PREFIX)/metricbeat-win.yml
rm $(PREFIX)/metricbeat-win.yml.bk
sed -i.bk 's/- load/#- load/' $(PREFIX)/metricbeat-win.full.yml
rm $(PREFIX)/metricbeat-win.full.yml.bk
endif
This issue seems to be introduced by dd43dd1
I tested with the two changes and the community beat gets properly packaged.
If you confirm these are valid issues I can submit separate github issues and pull requests for each one.
Thank you!
/F
Lets discuss / test these changes on a PR, then we see also if it does not break our packaging 