Custom filebeat processor

yoklmn · September 9, 2020, 2:37pm

Hello. I hard trying create my own plugin of filebeat's processor. I use filebeat v7.9.1 from official repository on machine B, CentOS 7.6.1810, CPU Intel Xeon E-2176G:
filebeat version 7.9.1 (amd64), libbeat 7.9.1 [ad823eca4cc74439d1a44351c596c12ab51054f5 built 2020-09-01 19:58:51 +0000 UTC]

On machine A, CentOS 7.5.1804, CPU Intel Core i7-3770 I run these commands:

yum install -y wget git gcc python python3
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum install -y docker-ce docker-ce-cli containerd.io
wget https://golang.org/dl/go1.14.7.linux-amd64.tar.gz #because here described ver. 1.14.17
tar -C /usr/local -xzf go1.14.7.linux-amd64.tar.gz
export PATH=$PATH:/usr/local/go/bin
export GOPATH=~/go
export PATH=$PATH:$GOPATH/bin

go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/root/go/src/github.com/elastic/beats/filebeat/processor/myplugin/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build129916806=/tmp/go-build -gno-record-gcc-switches"

go version
go version go1.14.7 linux/amd64

git clone GitHub - elastic/beats: 🐠 Beats - Lightweight shippers for Elasticsearch & Logstash ${GOPATH}/src/github.com/elastic/beats
cd $GOPATH/src/github.com/elastic/beats
git checkout ad823eca4cc74439d1a44351c596c12ab51054f5
cd libbeat
make

Then I create folder ${GOPATH}/src/github.com/elastic/beats/libbeat/processors/myplugin/ and put such files there:

${GOPATH}/src/github.com/elastic/beats/libbeat/processors/myplugin/main.go:

package main

import (
"github.com/elastic/beats/libbeat/plugin"
"github.com/elastic/beats/libbeat/processors"
)

var Bundle = plugin.Bundle(
processors.Plugin("myplugin", New),
)

and ${GOPATH}/src/github.com/elastic/beats/libbeat/processors/myplugin/myplugin.go:

package main

import (
"os"
"fmt"

"github.com/elastic/beats/libbeat/beat"
"github.com/elastic/beats/libbeat/common"
"github.com/elastic/beats/libbeat/processors"
)

type Myplugin struct {
}

func (f Myplugin) String() string {
return "Myplugin="
}

func New(c *common.Config) (processors.Processor, error) {
return &Myplugin{
}, nil
}

func (f *Myplugin) Run(event *beat.Event) (*beat.Event, error) {
l, err := event.GetValue("log.file.path")
if err != nil {
return nil, fmt.Errorf("fail getting log.file.path", err)
}

logfile := l.(string)

file, err := os.Stat(logfile)
modifiedtime := file.ModTime()

event.PutValue("event.log.file.timestamp", modifiedtime)
return event, nil
}

Then I run these commands:

cp $GOPATH/src/github.com/elastic/beats/go.mod $GOPATH/src/github.com/elastic/beats/libbeat/processors/myplugin/
cp $GOPATH/src/github.com/elastic/beats/go.sum $GOPATH/src/github.com/elastic/beats/libbeat/processors/myplugin/
go build -buildmode=plugin

Then I copy compiled plugin to machine B and run command:

/usr/bin/filebeat --plugin /etc/filebeat/myplugin.so

But I get these:

Exiting: plugin.Open("/etc/filebeat/myplugin"): plugin was built with a different version of package GitHub - pkg/errors: Simple error handling primitives

If I compile plugin with any of these commands:

go build -buildmode=plugin -trimpath
go mod vendor
GO111MODULE=on go build -mod=vendor -buildmode=plugin -trimpath
GO111MODULE=on go build -mod=vendor -buildmode=plugin

I get little different error:

Exiting: plugin.Open("/etc/filebeat/myplugin"): plugin was built with a different version of package internal/cpu

I tried other versions of Go, other branches of filebeat, even try compile filebeat on machine A and run compiled from sources filebeat with my plugin on machine B - nothing, same result.

Here it says the problem might be in GOPATH environment variable. I have value as /root/go, but I don't know, what value have GOPATH variable in official dev environment.

Please, help!