Beats Developer Guide [master] is out of date, creating new beat fails

There are errors in the documentation to get started with creating your own beats. If you search in the history, they have been around for some time and haven't been updated. This makes it extremely difficult to create a new beat, especially for someone new to GOlang.

I've cloned the beats repo per the developer setup description. Since I'm also developing against 7.10.1, I've checked out that version to my local branch to run the following...

  1. Page: Generating Your Beat
    The documentation says:

cd ${GOPATH}/src/github.com/{user}

Run the mage script to generate the custom beat:

mage GenerateCustomBeat

  • This is incorrect. To run "mage GenerateCustomBeat" you need to be in the beats directory that was cloned. But you need to create your user directory or GenerateCustomBeat fails. The documentation should be something like this:

mkdir -p ${GOPATH}/src/github.com/{user}
cd ${GOPATH}/src/github.com/src/github.com/elastic/beats
mage GenerateCustomBeat

  • This isn't in the documentation, but I had issues with GenerateCustomBeat failing while trying to run setup.GitAdd from the file: ${GOPATH}/src/github.com/src/github.com/elastic/beats/generator/common/beatgen/beatgen.go

  • I fixed this by commenting out the following line:

mg.Deps(setup.GitAdd)

  • It would be nice to explain in the documentation why it's trying do create repository and add the new beat.

  • Also on this page the user input for the GenerateCustomBeat is not the same and some explanations should be added to the documentation.

Enter the beat name [examplebeat]: <--- Should the be first initial capitalized?
Enter your github name [your-github-name]: <--- This should be the user specified above.
Enter the beat path [github.com/{user}/examplebeat] <-- github.com/{user} directory needs to exist
Enter your full name [Firstname Lastname]: <--- no issues
Enter the beat type [beat]: <--- What is this beat type referring to here?
Enter the GitHub - elastic/beats: 🐠 Beats - Lightweight shippers for Elasticsearch & Logstash revision [master]: <--- Is this the version tag or the git hash for the version you want to use, like 1da173a for v7.10.1?

  • If the beats revision is left as the default "master", the script fails:

# github.com/elastic/beats/v7/libbeat/common/transport/tlscommon
../../../../pkg/mod/github.com/elastic/beats/v7@v7.0.0-alpha2.0.20210203145817-10fdd24879c7/libbeat/common/transport/tlscommon/tls_config.go:112:3: unknown field 'VerifyConnection' in struct literal of type tls.Config
../../../../pkg/mod/github.com/elastic/beats/v7@v7.0.0-alpha2.0.20210203145817-10fdd24879c7/libbeat/common/transport/tlscommon/tls_config.go:123:4: unknown field 'VerifyConnection' in struct literal of type tls.Config
../../../../pkg/mod/github.com/elastic/beats/v7@v7.0.0-alpha2.0.20210203145817-10fdd24879c7/libbeat/common/transport/tlscommon/tls_config.go:141:4: unknown field 'VerifyConnection' in struct literal of type tls.Config
../../../../pkg/mod/github.com/elastic/beats/v7@v7.0.0-alpha2.0.20210203145817-10fdd24879c7/libbeat/common/transport/tlscommon/tls_config.go:149:8: config.VerifyConnection undefined (type *tls.Config has no field or method VerifyConnection)
note: module requires Go 1.15
Error: error compiling magefiles
make: *** [fields] Error 1
Error: running "make update" failed with exit code 2

  • I've gotten it to run by using v7.10.1 as the beats revision...
  1. Page: Fetching Dependencies and Setting
    The documentation says:

To fetch dependencies and set up the Beat, run:

cd ${GOPATH}/src/github.com/{user}/countbeat
make setup

  • This is incorrect, there is no target for setup, and was documented in this posting and should be:

make update

Now, when I run the 'mage build' command, I get the following output:

$ mage build
build: Building examplebeat
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git [...] -- [...]'
Error: template: inline:1:128: executing "inline" at : error calling commit: running "git rev-parse HEAD" failed with exit code 128
failed to expand template '-s -X github.com/elastic/beats/v7/libbeat/version.buildTime={{ date }} -X github.com/elastic/beats/v7/libbeat/version.commit={{ commit }}'
github.com/elastic/beats/v7/dev-tools/mage.expandTemplate
/Users/jackemuk/go/pkg/mod/github.com/elastic/beats/v7@v7.10.1/dev-tools/mage/common.go:100
github.com/elastic/beats/v7/dev-tools/mage.Expand
/Users/jackemuk/go/pkg/mod/github.com/elastic/beats/v7@v7.10.1/dev-tools/mage/common.go:56
github.com/elastic/beats/v7/dev-tools/mage.MustExpand
/Users/jackemuk/go/pkg/mod/github.com/elastic/beats/v7@v7.10.1/dev-tools/mage/common.go:62
github.com/elastic/beats/v7/dev-tools/mage.Build
/Users/jackemuk/go/pkg/mod/github.com/elastic/beats/v7@v7.10.1/dev-tools/mage/build.go:139
github.com/elastic/beats/v7/dev-tools/mage/target/build.Build
/Users/jackemuk/go/pkg/mod/github.com/elastic/beats/v7@v7.10.1/dev-tools/mage/target/build/build.go:26
main.Build
/Users/jackemuk/go/src/github.com/jackemuk-es/examplebeat/magefile.go:82
main.main.func14
/Users/jackemuk/go/src/github.com/jackemuk-es/examplebeat/mage_output_file.go:490
main.main.func12.1
/Users/jackemuk/go/src/github.com/jackemuk-es/examplebeat/mage_output_file.go:275
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1373

Does anyone from Elastic read these boards? Someone posted a similar issue and it was automatically closed because nobody responded: issue creating new beat #260339

What Go version are you using? It seems to me that you are trying to use an unsupported version of Go.

I was using 1.14.12 but upgraded to 1.15.7.

But the issue still remains the steps are not accurate or clear, such as that the command "make setup" does not exist, and there is no explanation in the documentation to explain what "Enter the beat type [beat]:" is referring to when running "mage GenerateCustomBeat"

The Beat type is beat when generating a new Beat. The document says that the proper value is selected by default. Alternatively, if you created a new metricset for Metricbeat, you should have set it to metricbeat.

Running make setup is not necessary anymore. You can just go ahead and run mage build.

I understand that, but please look at the title of this posting... the documentation is out of date. For someone new to elastic and beat development, this question is regularly asked because the documentation has not been updated to reflect that running make setup is not necessary.

The documentation should walk a user through all steps to produce a sample beat to run. The current documentation does not do this.

Thank you for the report. Do you mind opening an issue on GH or open a PR with the update?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.