GenerateCustomBeat returns error for missing github.com/elastic/beats/v7

Hi all,

I'm attempting to create a new custom beat but am hitting some errors when using the mage GenerateCustomBeat command.

mage GenerateCustomBeat
Enter the beat name [examplebeat]: testbeat
Enter your github name [your-github-name]: andrewexpanse
Enter the beat path [github.com/andrewexpanse/testbeat]:
Enter your full name [Firstname Lastname]: Andrew Scott
Enter the github.com/elastic/beats revision [master]:
go: creating new go.mod: module github.com/andrewexpanse/testbeat
go get github.com/elastic/beats/v7@master: module github.com/elastic/beats@master found (v7.0.0-alpha2.0.20200224095451-b0c600a28dbc+incompatible), but does not contain package github.com/elastic/beats/v7
Error: error while getting required beats version: running "go get -d -u github.com/elastic/beats/v7@master" failed with exit code 1

I did some searching but haven't seen any relevant topics yet. My dev environment is fresh after following these guidelines (https://www.elastic.co/guide/en/beats/devguide/current/beats-contributing.html#setting-up-dev-environment).

It seems like go is trying to fetch a too old version of master branch. We need to adjust the default value to make sure Go is not attempting to fetch an incorrect version. Do you mind opening an issue on GH?

You can work around the issue by providing the last commit from master.

Created https://github.com/elastic/beats/issues/17033

When you say I could work around this by providing the last commit from master, do you mean I should enter a commit hash/id when asked Enter the github.com/elastic/beats revision?

Exactly. Sorry for not being clear.

This could totally be user error - but I'm still seeing a very similar error when providing the last commit hash.

mage GenerateCustomBeat
Enter the beat name [examplebeat]: testbeat
Enter your github name [your-github-name]: andrewexpanse
Enter the beat path [github.com/andrewexpanse/testbeat]:
Enter your full name [Firstname Lastname]: Andrew Scott
Enter the beat type [beat]:
Enter the github.com/elastic/beats revision [master]: c06c106cabbd7252cf784f518c8188ef821913ea
go: creating new go.mod: module github.com/andrewexpanse/testbeat
go: downloading github.com/elastic/beats/v7 v7.0.0-alpha2.0.20200318142504-c06c106cabbd
go: github.com/elastic/beats/v7 c06c106cabbd7252cf784f518c8188ef821913ea => v7.0.0-alpha2.0.20200318142504-c06c106cabbd
go: golang.org/x/text upgrade => v0.3.2
go: gopkg.in/yaml.v2 upgrade => v2.2.8
go: golang.org/x/sys upgrade => v0.0.0-20200317113312-5766fd39f98d
go: golang.org/x/tools upgrade => v0.0.0-20200318132943-540150da7399
go: github.com/pkg/errors upgrade => v0.9.1
go: go.uber.org/atomic upgrade => v1.6.0
go: github.com/jstemmer/go-junit-report upgrade => v0.9.1
go: github.com/mitchellh/hashstructure upgrade => v1.0.0
go: github.com/prometheus/procfs upgrade => v0.0.10
go: github.com/go-sourcemap/sourcemap upgrade => v2.1.3+incompatible
go: github.com/elastic/go-windows upgrade => v1.0.1
go: github.com/elastic/go-ucfg upgrade => v0.8.3
go: go.uber.org/zap upgrade => v1.14.1
go: github.com/dop251/goja_nodejs upgrade => v0.0.0-20200128125109-2d688c7e0ac4
go: github.com/dop251/goja upgrade => v0.0.0-20200309191912-043cf4f34a48
go: go.uber.org/multierr upgrade => v1.5.0
go: github.com/dlclark/regexp2 upgrade => v1.2.0
go: github.com/josephspurrier/goversioninfo upgrade => v0.0.0-20200309025242-14b0ab84c6ca
go: howett.net/plist upgrade => v0.0.0-20200225050739-77e249a2e2ba
go: github.com/rcrowley/go-metrics upgrade => v0.0.0-20200313005456-10cdbea86bc0
go: downloading golang.org/x/tools v0.0.0-20200318132943-540150da7399
go: downloading golang.org/x/sys v0.0.0-20200317113312-5766fd39f98d
Error: expected 1 line, got 0 while looking for github.com/elastic/beats/v7

Probably not, a few parts of the generators are not working at the moment since we adapted go modules. Let me look into it.

What is your go version?

go1.14 darwin/amd64

The current supported Go version for beats is 1.13.8 as you can see in .go-version. Please use that to generate your new Beat. Also, that migth be broken as well. I am fixing that now.

The adoption of go 1.14 is tracked in this issue: https://github.com/elastic/beats/issues/16566

Thanks so much @kvch !

For anyone else who runs into this, if you used brew to install go here's a handy guide for downgrading. https://blog.notmyhostna.me/downgrade-go-installed-with-homebrew/

I have been trying for hours to get this to work and it is frustrating.

go version:

$ go version
go version go1.13.8 linux/amd64

when i try to generate with the last commit from master as revision, as mentioned above:

$ mage GenerateCustomBeat
Enter the beat name [examplebeat]: test
Enter your github name [your-github-name]: test
Enter the beat path [github.com/test/test]: 
Enter your full name [Firstname Lastname]: test
Enter the beat type [beat]: 
Enter the github.com/elastic/beats revision [master]: 70237a7b1ae1b36b9309278ee17085ea34cc032b
go: creating new go.mod: module github.com/test/test
go: finding github.com 70237a7b1ae1b36b9309278ee17085ea34cc032b
go: finding github.com/elastic/beats/v7 70237a7b1ae1b36b9309278ee17085ea34cc032b
go: finding github.com/elastic/beats 70237a7b1ae1b36b9309278ee17085ea34cc032b
go: finding github.com/elastic 70237a7b1ae1b36b9309278ee17085ea34cc032b
go: finding github.com/elastic/beats 70237a7b1ae1b36b9309278ee17085ea34cc032b
go: finding github.com/elastic/beats/v7 70237a7b1ae1b36b9309278ee17085ea34cc032b
go: finding github.com/dop251/goja_nodejs latest
go: finding github.com/dop251/goja latest
go: finding golang.org/x/sys latest
go: finding github.com/rcrowley/go-metrics latest
go: finding golang.org/x/tools latest
go: finding howett.net/plist latest
go: finding github.com/joeshaw/multierror latest
go: finding github.com/josephspurrier/goversioninfo latest
Cache entry deserialization failed, entry ignored
can't load package: package github.com/magefile/mage: no Go files in /home/johann/go/src/github.com/test/test/vendor/github.com/magefile/mage
make: *** [vendor/github.com/elastic/beats/v7/dev-tools/make/mage.mk:10: mage] Error 1
Error: running "make update" failed with exit code 2

I also tried different / older commits, as well as the branches 7.7, 7.6, .... nothing works

Did you get this to work @andrew-expanse ???

This is also effecting x-pack module builds.
mage
Error determining list of magefiles: failed to list non-mage gofiles: exit status 1: go: inconsistent vendoring in /root/go/src/github.com/elastic/beats:
go.mod requires github.com/elastic/beats/v7 but vendor/modules.txt does not include it.
run 'go mod tidy; go mod vendor' to sync

running provided go sync and tidy does not fix it as there is no v7 module available.

I did manage to get it to work after some trial and error, but I've since restarted my machine and lost the command that actually worked - I believe I was just doing what you were doing here though.

This is what I'm getting when trying to get the module code. Keep in mind I can build filebeat but not the x-pack modules:

go mod tidy; go mod vendor
warning: ignoring symlink /root/go/src/github.com/elastic/beats/build/python-env/lib64
warning: ignoring symlink /root/go/src/github.com/elastic/beats/libbeat/build/python-env/lib64
github.com/elastic/beats/v7 imports
	fmt: malformed module path "fmt": missing dot in first path element
github.com/elastic/beats/v7 imports
	os: malformed module path "os": missing dot in first path element
github.com/elastic/beats/v7 imports
	path/filepath: malformed module path "path/filepath": missing dot in first path element
github.com/elastic/beats/v7/auditbeat imports
	time: malformed module path "time": missing dot in first path element
github.com/elastic/beats/v7/auditbeat/datastore imports
	io: malformed module path "io": missing dot in first path element
github.com/elastic/beats/v7/auditbeat/datastore imports
	sync: malformed module path "sync": missing dot in first path element
github.com/elastic/beats/v7/auditbeat/helper/hasher imports
	crypto/md5: malformed module path "crypto/md5": missing dot in first path element
github.com/elastic/beats/v7/auditbeat/helper/hasher imports
	crypto/sha1: malformed module path "crypto/sha1": missing dot in first path element
github.com/elastic/beats/v7/auditbeat/helper/hasher imports
	crypto/sha256: malformed module path "crypto/sha256": missing dot in first path element
github.com/elastic/beats/v7/auditbeat/helper/hasher imports
	crypto/sha512: malformed module path "crypto/sha512": missing dot in first path element
github.com/elastic/beats/v7/auditbeat/helper/hasher imports
	encoding/hex: malformed module path "encoding/hex": missing dot in first path element
github.com/elastic/beats/v7/auditbeat/helper/hasher imports
	hash: malformed module path "hash": missing dot in first path element
github.com/elastic/beats/v7/auditbeat/helper/hasher imports
	strings: malformed module path "strings": missing dot in first path element
github.com/elastic/beats/v7/auditbeat/module/auditd imports
	bufio: malformed module path "bufio": missing dot in first path element
github.com/elastic/beats/v7/auditbeat/module/auditd imports
	bytes: malformed module path "bytes": missing dot in first path element
github.com/elastic/beats/v7/auditbeat/module/auditd imports
	os/user: malformed module path "os/user": missing dot in first path element
github.com/elastic/beats/v7/auditbeat/module/auditd imports
	runtime: malformed module path "runtime": missing dot in first path element
github.com/elastic/beats/v7/auditbeat/module/auditd imports
	sort: malformed module path "sort": missing dot in first path element
github.com/elastic/beats/v7/auditbeat/module/auditd imports
	strconv: malformed module path "strconv": missing dot in first path element
github.com/elastic/beats/v7/auditbeat/module/auditd imports
	syscall: malformed module path "syscall": missing dot in first path element
github.com/elastic/beats/v7/auditbeat/module/file_integrity imports
	errors: malformed module path "errors": missing dot in first path element
github.com/elastic/beats/v7/auditbeat/module/file_integrity imports
	math/bits: malformed module path "math/bits": missing dot in first path element
github.com/elastic/beats/v7/auditbeat/module/file_integrity imports
	sync/atomic: malformed module path "sync/atomic": missing dot in first path element
github.com/elastic/beats/v7/auditbeat/module/file_integrity/monitor imports
	path: malformed module path "path": missing dot in first path element
github.com/elastic/beats/v7/auditbeat/testing imports
	io/ioutil: malformed module path "io/ioutil": missing dot in first path element
github.com/elastic/beats/v7/auditbeat/testing imports
	testing: malformed module path "testing": missing dot in first path element
github.com/elastic/beats/v7/dev-tools/cmd/dashboards imports
	flag: malformed module path "flag": missing dot in first path element
github.com/elastic/beats/v7/dev-tools/cmd/dashboards imports
	log: malformed module path "log": missing dot in first path element
github.com/elastic/beats/v7/dev-tools/cmd/dashboards imports
	net/url: malformed module path "net/url": missing dot in first path element
github.com/elastic/beats/v7/dev-tools/cmd/license imports
	go/format: malformed module path "go/format": missing dot in first path element
github.com/elastic/beats/v7/dev-tools/cmd/license imports
	text/template: malformed module path "text/template": missing dot in first path element
github.com/elastic/beats/v7/dev-tools/mage imports
	archive/tar: malformed module path "archive/tar": missing dot in first path element
github.com/elastic/beats/v7/dev-tools/mage imports
	archive/zip: malformed module path "archive/zip": missing dot in first path element
github.com/elastic/beats/v7/dev-tools/mage imports
	compress/gzip: malformed module path "compress/gzip": missing dot in first path element
github.com/elastic/beats/v7/dev-tools/mage imports
	context: malformed module path "context": missing dot in first path element
github.com/elastic/beats/v7/dev-tools/mage imports
	encoding/json: malformed module path "encoding/json": missing dot in first path element
github.com/elastic/beats/v7/dev-tools/mage imports
	go/build: malformed module path "go/build": missing dot in first path element
github.com/elastic/beats/v7/dev-tools/mage imports
	net: malformed module path "net": missing dot in first path element
github.com/elastic/beats/v7/dev-tools/mage imports
	net/http: malformed module path "net/http": missing dot in first path element
github.com/elastic/beats/v7/dev-tools/mage imports
	os/exec: malformed module path "os/exec": missing dot in first path element
github.com/elastic/beats/v7/dev-tools/mage imports
	os/signal: malformed module path "os/signal": missing dot in first path element

etc etc