Packaging community beats


(Arnout Engelen) #1

Hello,

I'm having some trouble getting 'make package' to work with my community beat.

To rule out things specific to my beat, I've reproduced the problem with a freshly generated example beat. Below is a log of generating a beat using cookiecutter (under 'your-github-name' :slight_smile: ) and trying to package it:

➜  your-github-name$  cookiecutter $GOPATH/src/github.com/elastic/beats/generate/beat
project_name [Examplebeat]:
github_name [your-github-name]:
beat [examplebeat]:
beat_path [github.com/your-github-name]:
full_name [Firstname Lastname]:
➜  your-github-name$  cd examplebeat
➜  examplebeat$  make setup
mkdir -p vendor/github.com/elastic/
cp -R /home/aengelen/dev/go/src/github.com/elastic/beats vendor/github.com/elastic/
rm -rf vendor/github.com/elastic/beats/.git
make update
make[1]: Entering directory '/home/aengelen/dev/go/src/github.com/your-github-name/examp
lebeat'
Running virtualenv with interpreter /usr/bin/python2
New python executable in /home/aengelen/dev/go/src/github.com/your-github-name/examplebe
at/build/python-env/bin/python2
(....)
The index pattern was created under /home/aengelen/dev/go/src/github.com/your-github-nam
e/examplebeat/etc/kibana/index-pattern/examplebeat.json
make[1]: Leaving directory '/home/aengelen/dev/go/src/github.com/your-github-name/exampl
ebeat'
➜  examplebeat$  make
go build
➜  examplebeat$  make package
fatal: Not a git repository (or any parent up to mount point /home)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
echo "Start building packages for examplebeat"
Start building packages for examplebeat
mkdir -p /home/aengelen/dev/go/src/github.com/your-github-name/examplebeat/build/upload
if [ false = true ]; then \
	 make prepare-package-cgo; \
else \
	make prepare-package; \
fi
make[1]: Entering directory '/home/aengelen/dev/go/src/github.com/your-github-name/examp
lebeat'
fatal: Not a git repository (or any parent up to mount point /home)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
# cross compile on ubuntu
docker run --rm \
	-v /home/aengelen/dev/go/src/github.com/your-github-name/examplebeat/vendor/gith
ub.com/elastic/beats/dev-tools/packer/xgo-scripts:/scripts \
	-v /home/aengelen/dev/go/src/github.com/your-github-name:/source \
	-v /home/aengelen/dev/go/src/github.com/your-github-name/examplebeat/build:/buil
d \
	-e PUREGO="yes" \
	-e PACK=examplebeat \
	-e BEFORE_BUILD=before_build.sh \
	-e SOURCE=/source \
	-e TARGETS="linux/amd64 linux/386 windows/amd64 windows/386 darwin/amd64" \
	-e BUILDID= \
	tudorg/beats-builder \
	github.com/your-github-name/examplebeat
Working directory=/go/src/github.com/your-github-name
Copying main source folder /source to folder /go/src/github.com/your-github-name/examplebeat
Fetching dependencies...
Execute /scripts/before_build.sh github.com/your-github-name/examplebeat
make: *** No rule to make target `install-home'.  Stop.
vendor/github.com/elastic/beats/libbeat/scripts/Makefile:330: recipe for target 'prepare-package' failed
make[1]: *** [prepare-package] Error 2
make[1]: Leaving directory '/home/aengelen/dev/go/src/github.com/your-github-name/examplebeat'
vendor/github.com/elastic/beats/libbeat/scripts/Makefile:384: recipe for target 'package' failed
make: *** [package] Error 2

I've done some poking and it seems to me the problem might be in the docker volume for the beat sources. It now looks like:

-v /home/aengelen/dev/go/src/github.com/your-github-name:/source \

... so it mounts my complete github tree, instead of just the beat sources. Then inside the docker container it does expect just the beat in /source, nothing more:

Copying main source folder /source to folder /go/src/github.com/your-github-name/examplebeat

I've been looking into the libbeats makefiles for a bit, and it appears choosing to mount $(abspath ../) on /source in https://github.com/elastic/beats/blob/master/libbeat/scripts/Makefile#L351 might be the culprit. Could that perhaps be something like $GOPATH/src/$BEAT_DIR or shell pwd?


(Tudor Golubenco) #2

Something is strange, because I cannot reproduce this output. The /source/ should be copied to /go/src/github.com/your-github-name. Can you try deleting all your containers, the tudorg/beats-builder image and then the dangling images? I have a suspicion that Docker has cached an older version of the image.


(Arnout Engelen) #3

Ah - isn't that kind of excessive, as that means it'll copy all my github packages to the build system instead of just the beat I want to build? Anyway, if that's how it's intended to work I suppose it shouldn't hurt.

Hmm. I did:

docker ps -a | grep beats-builder | cut -d " " -f 1 | xargs docker rm
docker rmi tudorg/beats-builder
docker images -qf dangling=true | xargs docker rmi

But still:

➜  connbeat git:(packaging) ✗ make package

Makefile:45: warning: overriding recipe for target 'connb
eat'
vendor/github.com/elastic/beats/libbeat/scripts/Makefile:
73: warning: ignoring old recipe for target 'connbeat'
echo "Start building packages for connbeat"
Start building packages for connbeat
mkdir -p /home/aengelen/dev/go/src/github.com/raboof/conn
beat/build/upload
if [ false = true ]; then \
	 make prepare-package-cgo; \
else \
	make prepare-package; \
fi
make[1]: Entering directory '/home/aengelen/dev/go/src/gi
thub.com/raboof/connbeat'
Makefile:45: warning: overriding recipe for target 'connb
eat'
vendor/github.com/elastic/beats/libbeat/scripts/Makefile:
73: warning: ignoring old recipe for target 'connbeat'
# cross compile on ubuntu
docker run --rm \
	-v /home/aengelen/dev/go/src/github.com/raboof/co
nnbeat/vendor/github.com/elastic/beats/dev-tools/packer/x
go-scripts:/scripts \
	-v /home/aengelen/dev/go/src/github.com/raboof:/s
ource \
	-v /home/aengelen/dev/go/src/github.com/raboof/co
nnbeat/build:/build \
	-e PUREGO="yes" \
	-e PACK=connbeat \
	-e BEFORE_BUILD=before_build.sh \
	-e SOURCE=/source \
	-e TARGETS="linux/amd64 linux/386" \
	-e BUILDID=336a0df87c079b3db669f37e9ef6d3077b152a
ad \
	tudorg/beats-builder \
	github.com/raboof/connbeat
Unable to find image 'tudorg/beats-builder:latest' locall
y
latest: Pulling from tudorg/beats-builder
d89e1bee20d9: Pull complete
9e0bc8a71bde: Pull complete
27aa681c95e5: Pull complete
a3ed95caeb02: Pull complete
7e2842c4eae4: Pull complete
e5a6ee5b52a0: Pull complete
b839e3d921c4: Pull complete
a8f3c503d064: Pull complete
9e75498e242e: Pull complete
5dbf0bc0d74a: Pull complete
70488135beb6: Pull complete
570eabbcb9aa: Pull complete
667562fa680b: Pull complete
4e0f709f68e4: Pull complete
9e8557476d74: Pull complete
cc3e9ba40baa: Pull complete
4daf30f4027e: Pull complete
7856b1e716f5: Pull complete
5e25fdba8846: Pull complete
d41181cdd43b: Pull complete
f057b27a0baa: Pull complete
4f752490eba8: Pull complete
3cfbf6ae20f2: Pull complete
5c64b2ff812e: Pull complete
Digest: sha256:e1acdb23b9ccdcd722e9cb10b467f679b215d615d4b4dae753b6cf1d89a44db0
Status: Downloaded newer image for tudorg/beats-builder:latest
Working directory=/go/src/github.com/raboof
Copying main source folder /source to folder /go/src/github.com/raboof/connbeat
Fetching dependencies...
Execute /scripts/before_build.sh github.com/raboof/connbeat
make: *** No rule to make target `install-home'.  Stop.
vendor/github.com/elastic/beats/libbeat/scripts/Makefile:330: recipe for target 'prepare-package' failed
make[1]: *** [prepare-package] Error 2
make[1]: Leaving directory '/home/aengelen/dev/go/src/github.com/raboof/connbeat'
vendor/github.com/elastic/beats/libbeat/scripts/Makefile:384: recipe for target 'package' failed
make: *** [package] Error 2

I noticed the package I expect docker to be downloading is https://hub.docker.com/r/tudorg/beats-builder/tags/ which is from 5 months ago - does that sound accurate?


(ruflin) #4

Which version of the beats repository do you have locally checked out? Can you make sure to update it to the most recent version? Please also make sure to run make package-setup before running make package for the first time. This is hidden somewhere in the docs, but we should make this more obvious :frowning:

Side note: Just tried to create a new beat and run the setup and it worked. But obviously it could be that we have something on our dev machines that we kind of expect is everywhere but isn't.


(Tudor Golubenco) #5

That's not good. I'd expect it to build the images and only pull the OS image. I think make pacakge-setup will fix it for you, but I agree we should make that automatic.


(Tudor Golubenco) #6

We fixed the missing pacakge-setup in https://github.com/elastic/beats/pull/2668. Let us know if that fixes it for you.


(Arnout Engelen) #7

Ahhh, looks like make package-setup was indeed the trick! I think I ran it at some point, but it could very well be that I since removed the docker image again, causing the old version to get downloaded from dockerhub.

Perhaps we should remove that dockerhub image?

Adding package-setup as a Makefile dependency looks like a great solution, and indeed seems to make my beat build/package successfully now :slight_smile:.


(system) #8