File/Packet/Metric beat on Raspi Centos 7 (armhf/armv7l)

[m.pers]

Is there a build of packetbeat and filebeat for centos 7(armhf/armv7l arch) on the raspberry pi 3B+? I know raspbian has one but I am looking specifically for a build on Centos 7 with armv7l architecture or one close enough to port.

Release: CentOS Linux release 7.6.1810 (AltArch)
Kernel: 4.14.119-v7.1.el7
Arch: armv7l
Hw: Raspi 3B+

[davemoore]

@m.pers

There aren't official builds for Beats on ARM. But you can make them and they will work.

Recently I built Beats 7.1.0 on Raspberry Pi 3 B+ Ubuntu 18.04 64-bit ARMv8. Specifically I built Filebeat, Metricbeat, Auditbeat, and Heartbeat. Of those, I tested (and continue to run) Metricbeat and Auditbeat.

I'll walk through the build process that worked for me. It was a different OS (Ubuntu), a different version of ARM (v8), and I didn't try to build Packetbeat or run Filebeat. I won't try to adapt my steps to a platform that I haven't tested. But I expect this walkthrough will be similar enough that you can adjust the steps as needed, such as using yum instead of apt-get.

I look forward to hearing if these instructions help you build Beats on CentOS and ARMv7, and if Packetbeat and Filebeat work as expected on that platform.

Building Beats 7.1.0 on Ubuntu 18.04 (64-bit) ARMv8

Assume you have signed into the terminal as the ubuntu user.

Step 1: Install dependencies

sudo apt-get update
sudo apt-get install -y gcc git make wget

Step 2: Get Golang for ARM (64-bit)

Download Golang.

wget https://dl.google.com/go/go1.12.5.linux-arm64.tar.gz
tar -xzf go1.12.5.linux-arm64.tar.gz
mkdir go_projects

Export the new environment variables.

export GOROOT=$HOME/go
export PATH=$PATH:$GOROOT/bin
export GOPATH=$HOME/go_projects
export PATH=$GOPATH/bin:$PATH

Persist the new environment variables.

echo -e "export GOROOT=$HOME/go" | tee -a ~/.profile
echo -e "export PATH=$PATH:$GOROOT/bin" | tee -a ~/.profile
echo -e "export GOPATH=$HOME/go_projects" | tee -a ~/.profile
echo -e "export PATH=$PATH:$GOPATH/bin" | tee -a ~/.profile

Step 3: Get Beats

Get the source code for Beats from the official git repo, and then checkout the commit on which Beats 7.1.0 was built. If you want to build a specific version of Beats, then download the .tar.gz file for one of the Beats, extract it, and view the contents of the .build_hash file which shows the commit hash of that build. This build process was tested with the commit hash from metricbeat-7.1.0-linux-x86_64.tar.gz (03b3db2).

go get github.com/elastic/beats
cd $GOPATH/src/github.com/elastic/beats
git fetch
git checkout 03b3db2

Step 4: Build Beats

Temporarily enable swap space. The build process may require more memory than the capacity of the Raspberry Pi.

sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo -e "/swapfile swap swap defaults 0 0" | sudo tee -a /etc/fstab
sudo swapon --show

Make Metricbeat. An executable file named metricbeat will be created in $GOPATH/src/github.com/elastic/beats.

cd $GOPATH/src/github.com/elastic/beats/metricbeat
GOOS=linux GOARCH=arm64 go get
make

Make Auditbeat. An executable file named auditbeat will be created in $GOPATH/src/github.com/elastic/beats.

cd $GOPATH/src/github.com/elastic/beats/auditbeat
GOOS=linux GOARCH=arm64 go get
make

Make Filebeat. An executable file named filebeat will be created in $GOPATH/src/github.com/elastic/beats.

cd $GOPATH/src/github.com/elastic/beats/filebeat
GOOS=linux GOARCH=arm64 go get
make

Make Heartbeat. An executable file named heartbeat will be created in $GOPATH/src/github.com/elastic/beats.

cd $GOPATH/src/github.com/elastic/beats/heartbeat
GOOS=linux GOARCH=arm64 go get
make

Disable swap when done.

sudo swapoff -v /swapfile
# Remove the line below from /etc/fstab:
# /swapfile swap swap defaults 0 0
sudo rm /swapfile

Step 5: Organize Beats

Move each Beat to its own subdirectory under a single ~/beats directory.

cd ~
mkdir beats
cp -R $GOPATH/src/github.com/elastic/beats/auditbeat ~/beats/auditbeat
cp -R $GOPATH/src/github.com/elastic/beats/filebeat ~/beats/filebeat
cp -R $GOPATH/src/github.com/elastic/beats/heartbeat ~/beats/heartbeat
cp -R $GOPATH/src/github.com/elastic/beats/metricbeat ~/beats/metricbeat
cp $GOPATH/bin/auditbeat ~/beats/auditbeat
cp $GOPATH/bin/filebeat ~/beats/filebeat
cp $GOPATH/bin/heartbeat ~/beats/heartbeat
cp $GOPATH/bin/metricbeat ~/beats/metricbeat

Adjust the permissions and ownership for each Beat.

chmod go-w ~/beats/auditbeat/auditbeat.yml
chmod go-w ~/beats/filebeat/filebeat.yml
chmod go-w -R ~/beats/filebeat/modules.d
chmod go-w ~/beats/heartbeat/heartbeat.yml
chmod go-w ~/beats/metricbeat/metricbeat.yml
chmod go-w -R ~/beats/metricbeat/modules.d
sudo chown root:root ~/beats/auditbeat/auditbeat.yml

Step 6: Configure Beats

Configure each .yml file as you normally would.

Step 7: Setup Index Templates

Before running Beats for the first time, you should run the setup script to create the index templates and dashboards.

Our build process didn't package those index templates or dashboards. If you want to use this build to run the setup script, then download the official .tar.gz of each Beat and then copy its kibana directory into the right subdirectory of your ~/beats directory. After you do that, you can run commands such as ./metricbeat setup.

Step 8. Run Beats

We didn't build a convenient service script. But we can run Beats in the background using nohup.

For example, here's how to run Metricbeat in the background:

cd $HOME/beats/metricbeat && nohup ./metricbeat -e

Step 9. Start Beats on boot

You can schedule the command from Step 8 to run on boot using the crontab for the ubuntu user.

For example, here's how to schedule Metricbeat to run on boot:

(sudo crontab -u ubuntu -l; echo "@reboot . $HOME/.profile && cd $HOME/beats/metricbeat && nohup ./metricbeat -e &" ) | sudo crontab -u ubuntu -

Any Beats that must run as root, such as Auditbeat, need to be added to the crontab of the root user.

Proof

Here's an Elasticsearch cluster on six nodes of Raspberry Pi 3 B+ with each node running Metricbeat.

Cluster running over two availability zones:

Metricbeat system overview dashboard:

Beats monitoring overview:

Beats monitoring for one Beat:

[warkolm (Mark Walkom)]

Please also feel free to +1 this issue around support for ARM builds - https://github.com/elastic/beats/issues/9442

[m.pers]

Hi Dave,
thanks for the detailed write up! Centos7 on armv7l only comes in 32 bit, would similar steps suffice and would go from another source work? yum lists golang pkgs but none specifically just 'go'.

[davemoore]

@m.pers

While I haven't tested these steps to completion on 32-bit, I expect they will work after a couple adjustments. Beats supports 32-bit in general, given that the releases include 32-bit packages (example).

Here's what you need to change most likely:

• On Step 2, try using a different release of Golang, which are listed here. You'll need version 1.12.4 or higher as of the time of this post (source). Unfortunately I don't see a release specifically for arm7l. You could try using the release arm6l. But if it doesn't work, then you might need to build golang from source and then use that. For that build process, it's likely that you'll need to add some swap space temporarily given the memory it consumes.

• On Step 4, replace any instance of GOARCH=arm64 with GOARCH=arm (source).

Anecdotally, I tried building Beats on Raspbian 32-bit originally. I found the process of building Golang (see above) to be tedious, and it was one of the factors that led me to stick with the more widely used Ubuntu/CentOS distros with 64-bit support. But if you're comfortable with building things from source (personally I try to avoid it) then I expect this will work for you.

[andrewkroh]

@m.pers

Is there a build of packetbeat and filebeat for centos 7(armhf/armv7l arch) on the raspberry pi 3B+?

ARM is not an officially supported architecture because we don’t do any automated testing, but the build system supports generating both .deb and .rpm packages that target armv7. I use this when deploying to my rpi running raspian.

The build depends on Docker and Go (more details in the contributing guide). You need to have them both.

1. Git clone elastic/beats to $GOPATH/src/github.com/elastic/beats
2. git checkout v7.1.1 (checkout a release tag)
3. cd packetbeat
4. PLATFORMS="+all linux/armv7" make package
5. Wait while Packetbeat is cross-compiled inside a Docker container and then packaged into rpm/deb/tar.gz.
6. Look in build/distributions for the final outputs.

If you just want to test the packages and don't mind an unofficial SNAPSHOT build you can download one from our Jenkins CI servers. Like https://beats-ci.elastic.co/job/elastic+beats+7.2+multijob-package-linux/lastSuccessfulBuild/gcsObjects/.