Building packetbeat for the raspberry pi

Steinar_Bang · September 10, 2016, 6:35pm

I'm trying to build packagebeat for a raspberry Pi 2 with raspbian jessie.

I'm trying to build it on the rPi itself.

So far I've done :

As root installed required software (python 2.7.9, and GNU make was already installed)

apt-get install golang-go apt-get install python-virtualenv
2. As my own user, tried to build the 1.3 branch:

export GOPATH=$HOME/go mkdir -p $GOPATH/src/github.com/elastic cd $GOPATH/src/github.com/elastic git clone https://github.com/elastic/beats.git git checkout 1.3 cd beats/packetbeat make

However the make operation failed with the following output to stdout:

go build ../libbeat/scripts/Makefile:26: recipe for target 'build' failed

What was written to stderr by the same make command, was:

../libbeat/outputs/redis/redis.go:15:2: cannot find package "github.com/garyburd/redigo/redis" in any of: /usr/lib/go/src/pkg/github.com/garyburd/redigo/redis (from $GOROOT) /home/sb/go/src/github.com/garyburd/redigo/redis (from $GOPATH) ../libbeat/common/geolite.go:9:2: cannot find package "github.com/nranchev/go-libGeoIP" in any of: /usr/lib/go/src/pkg/github.com/nranchev/go-libGeoIP (from $GOROOT) /home/sb/go/src/github.com/nranchev/go-libGeoIP (from $GOPATH) protos/thrift/thrift_idl.go:8:2: cannot find package "github.com/samuel/go-thrift/parser" in any of: /usr/lib/go/src/pkg/github.com/samuel/go-thrift/parser (from $GOROOT) /home/sb/go/src/github.com/samuel/go-thrift/parser (from $GOPATH) protos/dns/dns.go:23:2: cannot find package "github.com/tsg/gopacket" in any of: /usr/lib/go/src/pkg/github.com/tsg/gopacket (from $GOROOT) /home/sb/go/src/github.com/tsg/gopacket (from $GOPATH) sniffer/afpacket_linux.go:9:2: cannot find package "github.com/tsg/gopacket/afpacket" in any of: /usr/lib/go/src/pkg/github.com/tsg/gopacket/afpacket (from $GOROOT) /home/sb/go/src/github.com/tsg/gopacket/afpacket (from $GOPATH) protos/tcp/tcp.go:12:2: cannot find package "github.com/tsg/gopacket/layers" in any of: /usr/lib/go/src/pkg/github.com/tsg/gopacket/layers (from $GOROOT) /home/sb/go/src/github.com/tsg/gopacket/layers (from $GOPATH) sniffer/sniffer.go:22:2: cannot find package "github.com/tsg/gopacket/pcap" in any of: /usr/lib/go/src/pkg/github.com/tsg/gopacket/pcap (from $GOROOT) /home/sb/go/src/github.com/tsg/gopacket/pcap (from $GOPATH) ../libbeat/cfgfile/cfgfile.go:12:2: cannot find package "gopkg.in/yaml.v2" in any of: /usr/lib/go/src/pkg/gopkg.in/yaml.v2 (from $GOROOT) /home/sb/go/src/gopkg.in/yaml.v2 (from $GOPATH) protos/mongodb/mongodb_parser.go:11:2: cannot find package "labix.org/v2/mgo/bson" in any of: /usr/lib/go/src/pkg/labix.org/v2/mgo/bson (from $GOROOT) /home/sb/go/src/labix.org/v2/mgo/bson (from $GOPATH) make: *** [build] Error 1

Is the problem that go is at version 1.3, and that I really need at least 1.5?

Is there a simple way to get where I want to go? Has someone else already built a .deb package that will install on the rPi?

Thanks!

Steinar

andrewkroh · September 10, 2016, 7:26pm

You are trying to build Packetbeat, correct? You wrote "packagebeat" which is a different project.

When Packetbeat 1.3 was released it was build with Go 1.7. So install that version and then Go should automatically find those missing dependencies which are included in the project's /vendor directory.

Steinar_Bang · September 10, 2016, 10:39pm

Yes, sorry. Packetbeat. I've changed the topic title.

Ok, There's a go1.7.1.linux-armv6l.tar.gz file in that directory, that might work. Worth a try, at least.

Debian jessie-backports has go 1.7, but finding backports for architectures other than i386 and/or amd64 seems futile (one might get lucky, but there are no guarantees they will ever get in place).

Unpacking a tarball and setting the PATH appropriately is probably a lot less work.

Thanks!

magnusbaeck · September 13, 2016, 5:55am

Unpacking a tarball and setting the PATH appropriately is probably a lot less work.

Or just build the binary on something other than your Raspberry Pi? The Go compiler is happy to cross-compile.

Steinar_Bang · September 15, 2016, 3:06pm

Magnus Bäck elastic@discoursemail.com:

Or just build the binary on something other than your Raspberry Pi? The Go compiler is happy to cross-compile.

Yes, but it's not just the compilation. I was hoping to be able to build a .deb package as well.

And I figured that would be simpler to do, if I did a native build on the rPi itself.

Steinar_Bang · September 17, 2016, 7:20pm

Now I've successfully built packetbeat on the rPi (well... there is a binary called packetbeat present, and I got no error message from the "make" command).

But I'm unable to figure out how to build a debian package. There is nothing about debian in the makefiles, as far as I can tell.

Is there stuff in the build setup to build a .deb of packagebeat?

Or would I have to look elsewhere for that? E.g. in debian somewhere?

magnusbaeck · September 17, 2016, 8:18pm

dev-tools/packer/platforms/debian/build.sh in the Beats git looks promising.

Steinar_Bang · September 17, 2016, 9:40pm

dev-tools/packer/platforms/debian/build.sh in the Beats git looks promising.

Hm... not present on the 1.3 branch...?
And it doesn't look like it can be easily transferred to that branch.

The reason for trying the 1.3 branch in the first place, was because
that was what I would have installed if I had been able to just pick a
.deb package and installing it.

However if 5.0 is what will build, that is what I will try...

Thanks again!

Steinar_Bang · September 17, 2016, 10:17pm

The command "docker" was missing:

[code]echo "Start building packages for packetbeat"
Start building packages for packetbeat
mkdir -p /home/sb/go/src/github.com/elastic/beats/packetbeat/build/upload
if [ true = true ]; then
make prepare-package-cgo;
else
make prepare-package;
fi
make[1]: Entering directory '/home/sb/go/src/github.com/elastic/beats/packetbeat'

cross compile on ubuntu

docker run --rm
-v /home/sb/go/src/github.com/elastic/beats/dev-tools/packer/xgo-scripts:/scripts
-v /home/sb/go/src/github.com/elastic/beats:/source
-v /home/sb/go/src/github.com/elastic/beats/packetbeat/build:/build
-e PACK=packetbeat
-e BEFORE_BUILD=before_build.sh
-e SOURCE=/source
-e TARGETS="windows/amd64 windows/386 darwin/amd64"
-e BUILDID=caeb4035662e2f591157f8cee2f3956bd09f5a15
tudorg/beats-builder
github.com/elastic/beats/packetbeat
bash: docker: command not found
../libbeat/scripts/Makefile:348: recipe for target 'prepare-package-cgo' failed
make[1]: *** [prepare-package-cgo] Error 127
make[1]: Leaving directory '/home/sb/go/src/github.com/elastic/beats/packetbeat'
../libbeat/scripts/Makefile:384: recipe for target 'package' failed
make: *** [package] Error 2[/code]

However if docker is the docker container, it isn't in debian jessie, this is the only package containing an executable named "docker":
https://packages.debian.org/jessie/docker

(and that is something other than the docker container...)

Steinar_Bang · September 17, 2016, 10:42pm

I found a package called docker.io which for regular debian only is in jessie-backports, but surprisingly was available in raspbian jessie (which doesn't have backports), and installed it on the rPi.

I tried doing "make package" as my own user but that failed since the command required privileges I don't have.

So I set GOROOT and GOPATH and added go to the PATH for root, and tried building, and ran into a new problem:

[code]root@ocon:/home/sb/go/src/github.com/elastic/beats/packetbeat# export GOROOT=/usr/local/go
root@ocon:/home/sb/go/src/github.com/elastic/beats/packetbeat# export GOPATH=/home/sb/go
root@ocon:/home/sb/go/src/github.com/elastic/beats/packetbeat# export PATH=$PATH:$GOROOT/bin
root@ocon:/home/sb/go/src/github.com/elastic/beats/packetbeat# make package
echo "Start building packages for packetbeat"
Start building packages for packetbeat
mkdir -p /home/sb/go/src/github.com/elastic/beats/packetbeat/build/upload
if [ true = true ]; then
make prepare-package-cgo;
else
make prepare-package;
fi
make[1]: Entering directory '/home/sb/go/src/github.com/elastic/beats/packetbeat'

cross compile on ubuntu

docker run --rm
-v /home/sb/go/src/github.com/elastic/beats/dev-tools/packer/xgo-scripts:/scripts
-v /home/sb/go/src/github.com/elastic/beats:/source
-v /home/sb/go/src/github.com/elastic/beats/packetbeat/build:/build
-e PACK=packetbeat
-e BEFORE_BUILD=before_build.sh
-e SOURCE=/source
-e TARGETS="windows/amd64 windows/386 darwin/amd64"
-e BUILDID=caeb4035662e2f591157f8cee2f3956bd09f5a15
tudorg/beats-builder
github.com/elastic/beats/packetbeat
Unable to find image 'tudorg/beats-builder' locally
Pulling repository tudorg/beats-builder
2016/09/18 00:34:51 Could not reach any registry endpoint
../libbeat/scripts/Makefile:348: recipe for target 'prepare-package-cgo' failed
make[1]: *** [prepare-package-cgo] Error 1
make[1]: Leaving directory '/home/sb/go/src/github.com/elastic/beats/packetbeat'
../libbeat/scripts/Makefile:384: recipe for target 'package' failed
make: *** [package] Error 2[/code]

Steinar_Bang · September 18, 2016, 6:12am

Googling the error message

it looks like I have an older version of docker.io than the docker image docker is trying to pull.

I've found the public image, but there was no version there that I could see: https://hub.docker.com/r/tudorg/beats-builder/

The version of docker.io that was installed on the rPi, was 1.3.3 :

# dpkg -l docker.io Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-========================-=================-=================-====================================================== ii docker.io 1.3.3~dfsg1-2 armhf Linux container runtime

ruflin · September 19, 2016, 7:47am

make package-setup is required before you run make package for the first time: https://github.com/elastic/beats/blob/master/libbeat/scripts/Makefile#L380

Not 100% sure if it will all build with your older docker version, but perhaps we will be lucky

Steinar_Bang · September 19, 2016, 3:42pm

Thanks for the tip! Didn't work unfortunately.

The output is a little different, but the core of the error message is still "Could not reach any registry endpoint" which google told me earlier meant a mismatch between the image and my docker.io version:/home/sb/go/src/github.com/elastic/beats/packetbeat# make package-setup make -C ../dev-tools/packer deps images make[1]: Entering directory '/home/sb/go/src/github.com/elastic/beats/dev-tools/packer' go get -u github.com/tsg/gotpl cd /home/sb/go/src/github.com/elastic/beats/dev-tools/packer/docker/xgo-image/; ./build.sh Sending build context to Docker daemon 318 kB Sending build context to Docker daemon Step 0 : FROM ubuntu:14.04 Pulling repository ubuntu 2016/09/19 17:37:58 Could not reach any registry endpoint Makefile:44: recipe for target 'xgo-image' failed make[1]: *** [xgo-image] Error 1 make[1]: Leaving directory '/home/sb/go/src/github.com/elastic/beats/dev-tools/packer' ../libbeat/scripts/Makefile:377: recipe for target 'package-setup' failed make: *** [package-setup] Error 2 /home/sb/go/src/github.com/elastic/beats/packetbeat#

Steinar_Bang · September 19, 2016, 3:51pm

On a cross-compiling note: If I try to cross compile on my regular computer, an Intel i5 (i.e. "amd64"), running debian jessie, will the build system be able to handle the dependency to pcap? I.e. to the libpcap.h file and presumably also a link with the native libpcap.so...?

Note that I was able to build the packetbeat binary itself natively on the rPi (at least from the 1.3 branch. I haven't actually tried building just the binary from 5.0 yet).

However, I needed to "apt-get install libpcap-dev" on the rPi to be able to build the binary, without the libpcap-dev package the packetbeat build complained a missing libpcap.h.

ruflin · September 20, 2016, 6:55am

Glad that you managed to compile it on your PI. About the compiling on a "regular computer". make package uses docker containers which already have all the dependencies inside for the compiling / packaging. If you do not compile it inside the docker container, I think you will also need to install libpcap.

system · October 1, 2016, 6:36pm

This topic was automatically closed after 21 days. New replies are no longer allowed.