GenerateCustomBeat returns error for missing github.com/elastic/beats/v7

andrew-expanse · March 16, 2020, 3:16pm

Hi all,

I'm attempting to create a new custom beat but am hitting some errors when using the mage GenerateCustomBeat command.

mage GenerateCustomBeat
Enter the beat name [examplebeat]: testbeat
Enter your github name [your-github-name]: andrewexpanse
Enter the beat path [github.com/andrewexpanse/testbeat]:
Enter your full name [Firstname Lastname]: Andrew Scott
Enter the github.com/elastic/beats revision [master]:
go: creating new go.mod: module github.com/andrewexpanse/testbeat
go get github.com/elastic/beats/v7@master: module github.com/elastic/beats@master found (v7.0.0-alpha2.0.20200224095451-b0c600a28dbc+incompatible), but does not contain package github.com/elastic/beats/v7
Error: error while getting required beats version: running "go get -d -u github.com/elastic/beats/v7@master" failed with exit code 1

I did some searching but haven't seen any relevant topics yet. My dev environment is fresh after following these guidelines (https://www.elastic.co/guide/en/beats/devguide/current/beats-contributing.html#setting-up-dev-environment).

kvch · March 16, 2020, 3:43pm

It seems like go is trying to fetch a too old version of master branch. We need to adjust the default value to make sure Go is not attempting to fetch an incorrect version. Do you mind opening an issue on GH?

You can work around the issue by providing the last commit from master.

andrew-expanse · March 17, 2020, 2:49pm

Created https://github.com/elastic/beats/issues/17033

When you say I could work around this by providing the last commit from master, do you mean I should enter a commit hash/id when asked Enter the github.com/elastic/beats revision?

kvch · March 17, 2020, 6:49pm

Exactly. Sorry for not being clear.

andrew-expanse · March 18, 2020, 4:23pm

This could totally be user error - but I'm still seeing a very similar error when providing the last commit hash.

mage GenerateCustomBeat
Enter the beat name [examplebeat]: testbeat
Enter your github name [your-github-name]: andrewexpanse
Enter the beat path [github.com/andrewexpanse/testbeat]:
Enter your full name [Firstname Lastname]: Andrew Scott
Enter the beat type [beat]:
Enter the github.com/elastic/beats revision [master]: c06c106cabbd7252cf784f518c8188ef821913ea
go: creating new go.mod: module github.com/andrewexpanse/testbeat
go: downloading github.com/elastic/beats/v7 v7.0.0-alpha2.0.20200318142504-c06c106cabbd
go: github.com/elastic/beats/v7 c06c106cabbd7252cf784f518c8188ef821913ea => v7.0.0-alpha2.0.20200318142504-c06c106cabbd
go: golang.org/x/text upgrade => v0.3.2
go: gopkg.in/yaml.v2 upgrade => v2.2.8
go: golang.org/x/sys upgrade => v0.0.0-20200317113312-5766fd39f98d
go: golang.org/x/tools upgrade => v0.0.0-20200318132943-540150da7399
go: github.com/pkg/errors upgrade => v0.9.1
go: go.uber.org/atomic upgrade => v1.6.0
go: github.com/jstemmer/go-junit-report upgrade => v0.9.1
go: github.com/mitchellh/hashstructure upgrade => v1.0.0
go: github.com/prometheus/procfs upgrade => v0.0.10
go: github.com/go-sourcemap/sourcemap upgrade => v2.1.3+incompatible
go: github.com/elastic/go-windows upgrade => v1.0.1
go: github.com/elastic/go-ucfg upgrade => v0.8.3
go: go.uber.org/zap upgrade => v1.14.1
go: github.com/dop251/goja_nodejs upgrade => v0.0.0-20200128125109-2d688c7e0ac4
go: github.com/dop251/goja upgrade => v0.0.0-20200309191912-043cf4f34a48
go: go.uber.org/multierr upgrade => v1.5.0
go: github.com/dlclark/regexp2 upgrade => v1.2.0
go: github.com/josephspurrier/goversioninfo upgrade => v0.0.0-20200309025242-14b0ab84c6ca
go: howett.net/plist upgrade => v0.0.0-20200225050739-77e249a2e2ba
go: github.com/rcrowley/go-metrics upgrade => v0.0.0-20200313005456-10cdbea86bc0
go: downloading golang.org/x/tools v0.0.0-20200318132943-540150da7399
go: downloading golang.org/x/sys v0.0.0-20200317113312-5766fd39f98d
Error: expected 1 line, got 0 while looking for github.com/elastic/beats/v7

kvch · March 18, 2020, 4:48pm

Probably not, a few parts of the generators are not working at the moment since we adapted go modules. Let me look into it.

kvch · March 18, 2020, 4:50pm

What is your go version?

andrew-expanse · March 18, 2020, 4:50pm

go1.14 darwin/amd64

kvch · March 18, 2020, 5:00pm

The current supported Go version for beats is 1.13.8 as you can see in .go-version. Please use that to generate your new Beat. Also, that migth be broken as well. I am fixing that now.

The adoption of go 1.14 is tracked in this issue: https://github.com/elastic/beats/issues/16566

andrew-expanse · March 18, 2020, 5:33pm

Thanks so much @kvch !

For anyone else who runs into this, if you used brew to install go here's a handy guide for downgrading. https://blog.notmyhostna.me/downgrade-go-installed-with-homebrew/

Johann_Tanzer · March 26, 2020, 8:55am

I have been trying for hours to get this to work and it is frustrating.

go version:

$ go version
go version go1.13.8 linux/amd64

when i try to generate with the last commit from master as revision, as mentioned above:

$ mage GenerateCustomBeat
Enter the beat name [examplebeat]: test
Enter your github name [your-github-name]: test
Enter the beat path [github.com/test/test]: 
Enter your full name [Firstname Lastname]: test
Enter the beat type [beat]: 
Enter the github.com/elastic/beats revision [master]: 70237a7b1ae1b36b9309278ee17085ea34cc032b
go: creating new go.mod: module github.com/test/test
go: finding github.com 70237a7b1ae1b36b9309278ee17085ea34cc032b
go: finding github.com/elastic/beats/v7 70237a7b1ae1b36b9309278ee17085ea34cc032b
go: finding github.com/elastic/beats 70237a7b1ae1b36b9309278ee17085ea34cc032b
go: finding github.com/elastic 70237a7b1ae1b36b9309278ee17085ea34cc032b
go: finding github.com/elastic/beats 70237a7b1ae1b36b9309278ee17085ea34cc032b
go: finding github.com/elastic/beats/v7 70237a7b1ae1b36b9309278ee17085ea34cc032b
go: finding github.com/dop251/goja_nodejs latest
go: finding github.com/dop251/goja latest
go: finding golang.org/x/sys latest
go: finding github.com/rcrowley/go-metrics latest
go: finding golang.org/x/tools latest
go: finding howett.net/plist latest
go: finding github.com/joeshaw/multierror latest
go: finding github.com/josephspurrier/goversioninfo latest
Cache entry deserialization failed, entry ignored
can't load package: package github.com/magefile/mage: no Go files in /home/johann/go/src/github.com/test/test/vendor/github.com/magefile/mage
make: *** [vendor/github.com/elastic/beats/v7/dev-tools/make/mage.mk:10: mage] Error 1
Error: running "make update" failed with exit code 2

I also tried different / older commits, as well as the branches 7.7, 7.6, .... nothing works

Did you get this to work @andrew-expanse ???

Mike_Ware · March 29, 2020, 5:28pm

This is also effecting x-pack module builds.
mage
Error determining list of magefiles: failed to list non-mage gofiles: exit status 1: go: inconsistent vendoring in /root/go/src/github.com/elastic/beats:
go.mod requires github.com/elastic/beats/v7 but vendor/modules.txt does not include it.
run 'go mod tidy; go mod vendor' to sync

running provided go sync and tidy does not fix it as there is no v7 module available.

andrew-expanse · March 29, 2020, 6:31pm

I did manage to get it to work after some trial and error, but I've since restarted my machine and lost the command that actually worked - I believe I was just doing what you were doing here though.

Mike_Ware · March 30, 2020, 4:23pm

This is what I'm getting when trying to get the module code. Keep in mind I can build filebeat but not the x-pack modules:

go mod tidy; go mod vendor
warning: ignoring symlink /root/go/src/github.com/elastic/beats/build/python-env/lib64
warning: ignoring symlink /root/go/src/github.com/elastic/beats/libbeat/build/python-env/lib64
github.com/elastic/beats/v7 imports
	fmt: malformed module path "fmt": missing dot in first path element
github.com/elastic/beats/v7 imports
	os: malformed module path "os": missing dot in first path element
github.com/elastic/beats/v7 imports
	path/filepath: malformed module path "path/filepath": missing dot in first path element
github.com/elastic/beats/v7/auditbeat imports
	time: malformed module path "time": missing dot in first path element
github.com/elastic/beats/v7/auditbeat/datastore imports
	io: malformed module path "io": missing dot in first path element
github.com/elastic/beats/v7/auditbeat/datastore imports
	sync: malformed module path "sync": missing dot in first path element
github.com/elastic/beats/v7/auditbeat/helper/hasher imports
	crypto/md5: malformed module path "crypto/md5": missing dot in first path element
github.com/elastic/beats/v7/auditbeat/helper/hasher imports
	crypto/sha1: malformed module path "crypto/sha1": missing dot in first path element
github.com/elastic/beats/v7/auditbeat/helper/hasher imports
	crypto/sha256: malformed module path "crypto/sha256": missing dot in first path element
github.com/elastic/beats/v7/auditbeat/helper/hasher imports
	crypto/sha512: malformed module path "crypto/sha512": missing dot in first path element
github.com/elastic/beats/v7/auditbeat/helper/hasher imports
	encoding/hex: malformed module path "encoding/hex": missing dot in first path element
github.com/elastic/beats/v7/auditbeat/helper/hasher imports
	hash: malformed module path "hash": missing dot in first path element
github.com/elastic/beats/v7/auditbeat/helper/hasher imports
	strings: malformed module path "strings": missing dot in first path element
github.com/elastic/beats/v7/auditbeat/module/auditd imports
	bufio: malformed module path "bufio": missing dot in first path element
github.com/elastic/beats/v7/auditbeat/module/auditd imports
	bytes: malformed module path "bytes": missing dot in first path element
github.com/elastic/beats/v7/auditbeat/module/auditd imports
	os/user: malformed module path "os/user": missing dot in first path element
github.com/elastic/beats/v7/auditbeat/module/auditd imports
	runtime: malformed module path "runtime": missing dot in first path element
github.com/elastic/beats/v7/auditbeat/module/auditd imports
	sort: malformed module path "sort": missing dot in first path element
github.com/elastic/beats/v7/auditbeat/module/auditd imports
	strconv: malformed module path "strconv": missing dot in first path element
github.com/elastic/beats/v7/auditbeat/module/auditd imports
	syscall: malformed module path "syscall": missing dot in first path element
github.com/elastic/beats/v7/auditbeat/module/file_integrity imports
	errors: malformed module path "errors": missing dot in first path element
github.com/elastic/beats/v7/auditbeat/module/file_integrity imports
	math/bits: malformed module path "math/bits": missing dot in first path element
github.com/elastic/beats/v7/auditbeat/module/file_integrity imports
	sync/atomic: malformed module path "sync/atomic": missing dot in first path element
github.com/elastic/beats/v7/auditbeat/module/file_integrity/monitor imports
	path: malformed module path "path": missing dot in first path element
github.com/elastic/beats/v7/auditbeat/testing imports
	io/ioutil: malformed module path "io/ioutil": missing dot in first path element
github.com/elastic/beats/v7/auditbeat/testing imports
	testing: malformed module path "testing": missing dot in first path element
github.com/elastic/beats/v7/dev-tools/cmd/dashboards imports
	flag: malformed module path "flag": missing dot in first path element
github.com/elastic/beats/v7/dev-tools/cmd/dashboards imports
	log: malformed module path "log": missing dot in first path element
github.com/elastic/beats/v7/dev-tools/cmd/dashboards imports
	net/url: malformed module path "net/url": missing dot in first path element
github.com/elastic/beats/v7/dev-tools/cmd/license imports
	go/format: malformed module path "go/format": missing dot in first path element
github.com/elastic/beats/v7/dev-tools/cmd/license imports
	text/template: malformed module path "text/template": missing dot in first path element
github.com/elastic/beats/v7/dev-tools/mage imports
	archive/tar: malformed module path "archive/tar": missing dot in first path element
github.com/elastic/beats/v7/dev-tools/mage imports
	archive/zip: malformed module path "archive/zip": missing dot in first path element
github.com/elastic/beats/v7/dev-tools/mage imports
	compress/gzip: malformed module path "compress/gzip": missing dot in first path element
github.com/elastic/beats/v7/dev-tools/mage imports
	context: malformed module path "context": missing dot in first path element
github.com/elastic/beats/v7/dev-tools/mage imports
	encoding/json: malformed module path "encoding/json": missing dot in first path element
github.com/elastic/beats/v7/dev-tools/mage imports
	go/build: malformed module path "go/build": missing dot in first path element
github.com/elastic/beats/v7/dev-tools/mage imports
	net: malformed module path "net": missing dot in first path element
github.com/elastic/beats/v7/dev-tools/mage imports
	net/http: malformed module path "net/http": missing dot in first path element
github.com/elastic/beats/v7/dev-tools/mage imports
	os/exec: malformed module path "os/exec": missing dot in first path element
github.com/elastic/beats/v7/dev-tools/mage imports
	os/signal: malformed module path "os/signal": missing dot in first path element

etc etc

nabeel_ahmed · April 14, 2020, 3:35pm

I am also facing the same issue on creating Custom beat
Go Version: 1.14.2

nabeel@na-08:~/go/src/github.com/elastic/beats$ NEWBEAT_TYPE=metricbeat mage GenerateCustomBeat
    Enter the beat name [examplebeat]: check1
    Enter your github name [your-github-name]: nabeel08
    Enter the beat path [github.com/nabeel08/check1]: 
    Enter your full name [Firstname Lastname]: N n
    Enter the github.com/elastic/beats revision [master]: master
    go: creating new go.mod: module github.com/nabeel08/check1
    go: downloading github.com/elastic/beats/v7 v7.0.0-alpha2.0.20200414143427-07ed0dabcaab
    go: github.com/elastic/beats/v7 master => v7.0.0-alpha2.0.20200414143427-07ed0dabcaab
    go: github.com/mitchellh/hashstructure upgrade => v1.0.0
    go: github.com/pkg/errors upgrade => v0.9.1
    go: gopkg.in/yaml.v2 upgrade => v2.2.8
    go: github.com/go-sourcemap/sourcemap upgrade => v2.1.3+incompatible
    go: go.uber.org/zap upgrade => v1.14.1
    go: github.com/dop251/goja upgrade => v0.0.0-20200414142002-77e84ffb8c65
    go: github.com/dop251/goja_nodejs upgrade => v0.0.0-20200128125109-2d688c7e0ac4
    go: golang.org/x/tools upgrade => v0.0.0-20200414131530-0037cb7812fa
    go: github.com/elastic/go-ucfg upgrade => v0.8.3
    go: github.com/rcrowley/go-metrics upgrade => v0.0.0-20200313005456-10cdbea86bc0
    go: howett.net/plist upgrade => v0.0.0-20200225050739-77e249a2e2ba
    go: golang.org/x/sys upgrade => v0.0.0-20200413165638-669c56c373c4
    go: go.uber.org/multierr upgrade => v1.5.0
    go: golang.org/x/text upgrade => v0.3.2
    go: github.com/prometheus/procfs upgrade => v0.0.11
    go: github.com/elastic/go-windows upgrade => v1.0.1
    go: github.com/jstemmer/go-junit-report upgrade => v0.9.1
    go: github.com/dlclark/regexp2 upgrade => v1.2.0
    go: go.uber.org/atomic upgrade => v1.6.0
    go: github.com/josephspurrier/goversioninfo upgrade => v0.0.0-20200309025242-14b0ab84c6ca
    Error: error while copying required files to vendor: error while looking up cached dir of module: github.com/elastic/beats/v7: expected 1 line, got 0 while looking for github.com/elastic/beats/v7

Is there any workaround to solve this ?

whatgeorgemade · April 15, 2020, 9:00pm

I managed to get it working by using a previous commit SHA. When asked for the revision of beats to use, don't just hit enter; paste in the SHA of the commit previous to the most recent in master, which is currently 52fa26598cef8a90c8981970414c780e4fd31dbf.

You might also have to downgrade to Go 1.13.8.

nabeel_ahmed · April 16, 2020, 6:55am

@ whatgeorgemade Downgrading works for me.

whatgeorgemade · April 16, 2020, 10:06pm

Can you explain what the v7 part of the repository is? I can't find a branch or tag with that name but see it everywhere in the code. Why was this added?

I'm working on a PR that is based on a branch from master but now can't use it in a beat because it pulls beats/v7 into the vendors folder.

EDIT: After reading up on Go modules I figured out that adding a replace to go.mod would suffice. After doing that and running mage vendorUpdate, I can test my branch.

kvch · April 20, 2020, 8:53am

Go modules require us to add the major version to the import path:

By definition, a new major version of a package is not backwards compatible with the previous version. This means a new major version of a module must have a different module path than the previous version. Starting with v2 , the major version must appear at the end of the module path (declared in the module statement in the go.mod file). For example, when the authors of the module github.com/googleapis/gax-go developed v2 , they used the new module path github.com/googleapis/gax-go/v2 . Users who wanted to use v2 had to change their package imports and module requirements to github.com/googleapis/gax-go/v2 .

Ref: Go Modules: v2 and Beyond - The Go Programming Language

Szanik · May 5, 2020, 10:19am

Is there a better fix outhere?

I have the same problem.
Ttried to downgrade my go version to 1.13.8
and tried the commit that "whatgeorgemade" mentioned.
And then again with the master.

nothing worked.
It got suck on some the go: finding
e.g.
go: finding github.com/josephspurrier/goversioninfo latest

The problem is also that, at this point of writing, there are 541 Powershell process because of
that.

Topic		Replies	Views
Error creating new beat Beats beats-development	2	1136	May 9, 2020
GenerateCustomBeats hangs Beats beats-development	1	474	March 18, 2021
Beats Developer Guide [master] is out of date, creating new beat fails Beats beats-development	6	1066	March 8, 2021
GenerateCustomBeat Beats	2	458	March 24, 2020
Unable to build custom beat Beats beats-development	5	1149	October 8, 2020

GenerateCustomBeat returns error for missing github.com/elastic/beats/v7

Related topics