Build filebeat x-pack modules from source fails

Mike_Ware · April 1, 2020, 9:33pm

I have managed to build file beat and base modules from source found at https://github.com/elastic/beats . When I try to build the modules in beats/x-pack/filebeat I receive the following error with build launched at src/github.com/elastic/beats/x-pack/filebeat

mage build
>> build: Building filebeat
../libbeat/cmd/inject.go:17:2: build constraints exclude all Go files in /root/go/src/github.com/elastic/beats/x-pack/libbeat/processors/add_cloudfoundry_metadata
Error: running "go build -o filebeat -ldflags -s -X github.com/elastic/beats/v7/libbeat/version.buildTime=2020-04-01T21:26:46Z -X github.com/elastic/beats/v7/libbeat/version.commit=0ef472268ea41881f81accabe2af6cfb72eef682" failed with exit code 1

I have little experience with Go and so am lost as far as troubleshooting this goes. The one things that I notice is the reference to the github.com/elastic/beats/v7 which does not exists in the source tree.

go version go1.14.1 freebsd/amd64

go env : 
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="freebsd"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="freebsd"
GOPATH="/root/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/freebsd_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/root/go/src/github.com/elastic/beats/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build279549248=/tmp/go-build -gno-record-gcc-switches"

Thanks in advance for any help.

ChrsMark · April 2, 2020, 8:40am

Hi @Mike_Ware!

I was not able to reproduce this one:

~/go/src/github.com/elastic/beats/x-pack/filebeat  git log --pretty=oneline                                                                              

495595d6240881661e9b15059f3fa70a5c400edc (HEAD -> master, upstream/master, origin/master) Remove dataType field of query metricset (#17383)
9009938d64c3f2c688b2b8ecd6afbb7108d0ddc3 Add pattern to handle logs from newer ES versions (#17406)
e3c72b308f6d4e3290690365aa7adb6939608dec [Filebeat] Improve ECS categorization field mappings for mssql module (#17376)
e7229a4d36de8bb55e5c41958374e064b9ea38fe [Filebeat] Improve ECS categorization field mappings for mongodb module (#17371)

And:

~/go/src/github.com/elastic/beats/x-pack/filebeat mage build                                                                                            
>> build: Building filebeat
 ~/go/src/github.com/elastic/beats/x-pack/filebeat  ./filebeat version                                                                                               
filebeat version 8.0.0 (amd64), libbeat 8.0.0 [495595d6240881661e9b15059f3fa70a5c400edc built 2020-04-02 08:39:10 +0000 UTC]

What could be different in your case?

Mike_Ware · April 2, 2020, 4:54pm

Hi Chris thanks for your reply. I am on freebsd 11.3 which could be a difference. Is there a solid set of instructions for setting up the build environment? I feel like I may just be missing something simple that's killing me.

Mike_Ware · April 2, 2020, 11:57pm

I just tried on a pretty fresh freebsd 12 system and it failed as well. Here's the entire process I followed.

Install python 3.8

Link python

ln -s /usr/local/bin/python3.8 /usr/local/bin/python

ln -s /usr/local/bin/python3.8 /usr/local/bin/python3

Install latest go
wget https://dl.google.com/go/go1.14.1.freebsd-386.tar.gz

tar -C /usr/local -xzf go1.14.1.freebsd-386.tar.gz

Add paths to profiles for execution

Install mage

go get -u -d GitHub - magefile/mage: a Make/rake-like dev tool using Go cd $GOPATH/src/github.com/magefile/mage go run bootstrap.go

Clone beats repo
Create /root/go/src/github.com/elastic folder and clone beats

git clone GitHub - elastic/beats: 🐠 Beats - Lightweight shippers for Elasticsearch & Logstash

Build filebeat

~/go/src/github.com/elastic/beats/filebeat # mage build
Succeeds

This is where we fail

~/go/src/github.com/elastic/beats/x-pack/filebeat # mage build

build: Building filebeat
../libbeat/cmd/inject.go:17:2: build constraints exclude all Go files in /root/go/src/github.com/elastic/beats/x-pack/libbeat/processors/add_cloudfoundry_metadata
Error: running "go build -o filebeat -ldflags -s -X github.com/elastic/beats/v7/libbeat/version.buildTime=2020-04-02T16:42:12Z -X github.com/elastic/beats/v7/libbeat/version.commit=ee1d6f57dffc73d579a22e0ea330cc6ec3bf34ce" failed with exit code 1

blaker · April 3, 2020, 1:07pm

@Mike_Ware Could you try to remove both of these lines to see if it will build?

github.com

elastic/beats/blob/aa13b468b71ec713b3f0aebaf813b57caf90a706/x-pack/libbeat/processors/add_cloudfoundry_metadata/add_cloudfoundry_metadata.go#L5


// Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
// or more contributor license agreements. Licensed under the Elastic License;
// you may not use this file except in compliance with the Elastic License.


// +build linux darwin windows


package add_cloudfoundry_metadata


import (
	"time"


	"github.com/pkg/errors"


	"github.com/elastic/beats/v7/libbeat/beat"
	"github.com/elastic/beats/v7/libbeat/common"

github.com

elastic/beats/blob/aa13b468b71ec713b3f0aebaf813b57caf90a706/x-pack/libbeat/processors/add_cloudfoundry_metadata/add_cloudfoundry_metadata_test.go#L5


// Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
// or more contributor license agreements. Licensed under the Elastic License;
// you may not use this file except in compliance with the Elastic License.


// +build linux darwin windows


package add_cloudfoundry_metadata


import (
	"fmt"
	"testing"
	"time"


	"github.com/cloudfoundry-community/go-cfclient"
	"github.com/gofrs/uuid"

I think those lines might have been added when they are really not required for cloudfoundry.

Mike_Ware · April 3, 2020, 4:02pm

@blaker Thanks for the tip, but neither of those files exist in my source tree. In fact there is no blob folder at all. find /root/go/src/ -name blob returns nothing and the metadata files are not in the path you proposed. Did I miss a step in generating?

I did find it in the libbeat x-pack tree.

find /root/go/src/ -name add_cloudfoundry_metadata.go

/root/go/src/github.com/elastic/beats/x-pack/libbeat/processors/add_cloudfoundry_metadata/add_cloudfoundry_metadata.go

Thanks for your help.

Mike_Ware · April 3, 2020, 4:42pm

Also interesting is it looks as though the automated build is failing : https://travis-ci.org/github/elastic/beats/jobs/670282371

blaker · April 3, 2020, 5:29pm

@Mike_Ware The job you linked to shows passed.

Yes you need to update it in that path that you found with find. Where you able to get it to build with that line removed?

Mike_Ware · April 3, 2020, 5:52pm

@blaker Ah strange it was showing as failing when I posted. And yes removing that line has fixed the build. Thanks so much for your help I thought I was going crazy or just missing something very simple!

Mike_Ware · April 3, 2020, 6:14pm

@blaker one last question, Is there a way to cleanly install from source? I've tried make/mage/gmake install with no luck.

blaker · April 3, 2020, 6:40pm

@Mike_Ware mage build in the directory should produce an output in build/. From there you can just use the binary, can move it to /usr/local/bin if you would like.

I have a fix going in to remove those lines so you do not need the modifications.

Mike_Ware · April 3, 2020, 7:58pm

@blaker thanks much appreciated. I figured just moving was the easy way just wondering if I was missing something else in the build.

system · May 1, 2020, 7:58pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.