Hello,
I am trying to create a new beat based on metricbeat. I have followed the instructions and am able to build and debug my new beat. All works as expected.
However, when I try to create a package using "make package" as is instructed in the developer docs, the process hangs. The only error generated during the process is related to kibana, however the message says the step is being ignored.
Any help would be appreciated.
Thanks
kenrowland,
Do you mean mage package ? I don't think we have a package make role.
If so, can you re-run it with the debug flag:
mage -debug package
It might also be worth updating your vendor directory:
# Pull down the latest version of elastic/beats into your gopath, then in your custom beat directory, run this
rm -r vendor/
make copy-vendor
Alex,
Thank you for the reply. This is my first attempt to create a beat based on metricbeat. In attempting to create the package, as is stated in the documentation, I actually ran the command "make release", not "make package" as is stated in my original post. I am sorry for the confusion. To be clear, when I run "make release" the process hangs.
Any help is appreciated as I expect to need to create 3-4 additional beats for our open source cluster project. The project is HPCCSystems should you be interested. It is a big data platform for both crunching and delivering data.
Ah, so it turns out that make release is just an alias for mage package.
A few things:
If you run mage update does that operation complete?
You can use the PLATFORMS variable to only build for certain platforms. Depending on where we're stuck, this might speed things up. Example: export PLATFORMS=linux
What "type" of custom beat is this? It's possible for users to specify --type=beat and --type=metricbeat when they create a beat. If you don't remember, just run this in your beat directory and send me the output: grep -A 5 setup Makefile
If mage update doesn't complete, send me the output of mage -debug update. If update works, try mage -debug package
Alex,
Again, thank you for your help.
The beat was created as a beat based on metricbeat. I followed the instructions from this page: https://www.elastic.co/guide/en/beats/devguide/current/creating-beat-from-metricbeat.html
I added PLATFORMS=linux and it had no effect. mage update completes w/o any problems. When I make my beat I am able to run it in debug mode and it works as expected.
Running 'mage package' still hangs, so I ran 'mage -debug package'. The last lines of the output are copied below.
DEBUG: 13:38:47.918438 found target PackageTest
DEBUG: 13:38:47.918481 time parse Magefiles: 329.312µs
DEBUG: 13:38:47.918496 setting alias "" and package "pkg" on func PackageTest
DEBUG: 13:38:48.028607 parsing imported package "github.com/elastic/beats/dev-tools/mage/target/test" from dir "/home/rowlke01/go/src/github.com/elastic/beats/dev-tools/mage/target/test"
DEBUG: 13:38:48.028961 skipping function with invalid signature func RegisterDeps(deps &{1008 0xc0001f72a0})()
DEBUG: 13:38:48.029012 found target Test
DEBUG: 13:38:48.029051 time parse Magefiles: 408.508µs
DEBUG: 13:38:48.029070 setting alias "" and package "test" on func Test
DEBUG: 13:38:48.216537 parsing imported package "github.com/elastic/beats/dev-tools/mage/target/unittest" from dir "/home/rowlke01/go/src/github.com/elastic/beats/dev-tools/mage/target/unittest"
DEBUG: 13:38:48.216717 found target GoUnitTest
DEBUG: 13:38:48.216732 found target PythonUnitTest
DEBUG: 13:38:48.216742 skipping function with invalid signature func RegisterGoTestDeps(deps &{1173 0xc000210420})()
DEBUG: 13:38:48.216745 skipping function with invalid signature func RegisterPythonTestDeps(deps &{1348 0xc0002105e0})()
DEBUG: 13:38:48.216747 found target UnitTest
DEBUG: 13:38:48.216754 time parse Magefiles: 203.019µs
DEBUG: 13:38:48.216760 setting alias "" and package "unittest" on func GoUnitTest
DEBUG: 13:38:48.216762 setting alias "" and package "unittest" on func PythonUnitTest
DEBUG: 13:38:48.216764 setting alias "" and package "unittest" on func UnitTest
DEBUG: 13:38:48.381380 parsing imported package "github.com/elastic/beats/dev-tools/mage/target/integtest" from dir "/home/rowlke01/go/src/github.com/elastic/beats/dev-tools/mage/target/integtest"
DEBUG: 13:38:48.381602 found target GoIntegTest
DEBUG: 13:38:48.381616 found target IntegTest
DEBUG: 13:38:48.381621 found target PythonIntegTest
DEBUG: 13:38:48.381632 skipping function with invalid signature func RegisterGoTestDeps(deps &{1213 0xc000211ea0})()
DEBUG: 13:38:48.381637 skipping function with invalid signature func RegisterPythonTestDeps(deps &{1389 0xc000232040})()
DEBUG: 13:38:48.381640 skipping function with invalid signature func WhitelistEnvVar(key &{1623 string})()
DEBUG: 13:38:48.381645 time parse Magefiles: 250.382µs
DEBUG: 13:38:48.381651 setting alias "" and package "integtest" on func GoIntegTest
DEBUG: 13:38:48.381653 setting alias "" and package "integtest" on func IntegTest
DEBUG: 13:38:48.381655 setting alias "" and package "integtest" on func PythonIntegTest
DEBUG: 13:38:48.381679 Creating mainfile at mage_output_file.go
DEBUG: 13:38:48.381713 writing new file at mage_output_file.go
DEBUG: 13:38:48.383481 compiling to build/mage-linux-amd64
DEBUG: 13:38:48.383490 compiling using gocmd: go
DEBUG: 13:38:48.442794 running go build -o build/mage-linux-amd64 magefile.go mage_output_file.go
DEBUG: 13:38:48.792996 time to compile Magefile: 350.145721ms
Thanks.
Something seems fishy here. What version of mage is this? You can check with mage -version It might also be worth installing whatever the newest release is. Also try running mage -clean
Also, try mage -v package. The output should be more helpful than -debug and should allow you to paste the whole thing.
Alex,
As requested:
mage -v shows
Mage Build Tool v1.8.0
Build Date:
Commit:
built with: go1.12.7
mage-v package output:
2019/09/11 14:46:01 Found Elastic Beats dir at /home/rowlke01/go/src/github.com/kenrowland/roxiebeat/vendor/github.com/elastic/beats
Running target: Package
community_beat package spec loaded from [/home/rowlke01/go/src/github.com/kenrowland/roxiebeat/vendor/github.com/elastic/beats/dev-tools/packaging/packages.yml]
Running dependency: Update
exec: make update
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
2019/09/11 14:46:10 Found Elastic Beats dir at /home/rowlke01/go/src/github.com/kenrowland/roxiebeat/vendor/github.com/elastic/beats
Running target: Fields
exec: go run /home/rowlke01/go/src/github.com/kenrowland/roxiebeat/vendor/github.com/elastic/beats/libbeat/scripts/cmd/global_fields/main.go -es_beats_path /home/rowlke01/go/src/github.com/kenrowland/roxiebeat/vendor/github.com/elastic/beats -beat_path /home/rowlke01/go/src/github.com/kenrowland/roxiebeat -out fields.yml module
Generated fields.yml for roxiebeat to /home/rowlke01/go/src/github.com/kenrowland/roxiebeat/fields.yml
go run ./vendor/github.com/elastic/beats/metricbeat/scripts/assets/assets.go ./vendor/github.com/elastic/beats/metricbeat/module
mkdir -p include/fields
go run ./vendor/github.com/elastic/beats/libbeat/scripts/cmd/global_fields/main.go -es_beats_path ./vendor/github.com/elastic/beats -beat_path /home/rowlke01/go/src/github.com/kenrowland/roxiebeat | go run ./vendor/github.com/elastic/beats/dev-tools/cmd/asset/asset.go -license ASL2 -out ./include/fields/fields.go -pkg include -priority asset.LibbeatFieldsPri ./vendor/github.com/elastic/beats/libbeat/fields.yml roxiebeat
mage CollectAll
2019/09/11 14:46:15 Found Elastic Beats dir at /home/rowlke01/go/src/github.com/kenrowland/roxiebeat/vendor/github.com/elastic/beats
Running target: CollectAll
Running dependency: FieldsDocs
exec: go run /home/rowlke01/go/src/github.com/kenrowland/roxiebeat/vendor/github.com/elastic/beats/libbeat/scripts/cmd/global_fields/main.go -es_beats_path /home/rowlke01/go/src/github.com/kenrowland/roxiebeat/vendor/github.com/elastic/beats -beat_path /home/rowlke01/go/src/github.com/kenrowland/roxiebeat -out build/fields/fields.all.yml /home/rowlke01/go/src/github.com/kenrowland/roxiebeat/module
Running dependency: CollectDocs
Generated fields.yml for roxiebeat to /home/rowlke01/go/src/github.com/kenrowland/roxiebeat/build/fields/fields.all.yml
>> Generating docs/fields.asciidoc for roxiebeat
exec: /home/rowlke01/go/src/github.com/kenrowland/roxiebeat/build/ve/linux/bin/python /home/rowlke01/go/src/github.com/kenrowland/roxiebeat/vendor/github.com/elastic/beats/libbeat/scripts/generate_fields_docs.py build/fields/fields.all.yml roxiebeat /home/rowlke01/go/src/github.com/kenrowland/roxiebeat/vendor/github.com/elastic/beats --output_path /home/rowlke01/go/src/github.com/kenrowland/roxiebeat
grep: ../libbeat/docs/version.asciidoc: No such file or directory
/home/rowlke01/go/src/github.com/kenrowland/roxiebeat/build/python-env/bin/python ./vendor/github.com/elastic/beats/script/modules_collector.py --beat roxiebeat --docs_branch=
cp: cannot stat 'module/*/_meta/kibana/*': No such file or directory
vendor/github.com/elastic/beats/metricbeat/Makefile:19: recipe for target 'kibana' failed
make: [kibana] Error 1 (ignored)
2019/09/11 14:46:21 Found Elastic Beats dir at /home/rowlke01/go/src/github.com/kenrowland/roxiebeat/vendor/github.com/elastic/beats
Running target: Config
>> Building roxiebeat.yml for linux/amd64
>> Building roxiebeat.reference.yml for linux/amd64
>> Building roxiebeat.docker.yml for linux/amd64
Updating generated files for roxiebeat
mkdir -p include
go run ./vendor/github.com/elastic/beats/dev-tools/cmd/asset/asset.go -license ASL2 -pkg include -in fields.yml -out include/fields.go roxiebeat
if [ -d /home/rowlke01/go/src/github.com/kenrowland/roxiebeat/_meta/kibana ]; then \
cp -pr /home/rowlke01/go/src/github.com/kenrowland/roxiebeat/_meta/kibana/* /home/rowlke01/go/src/github.com/kenrowland/roxiebeat/_meta/kibana.generated ; \
fi
Running dependency: github.com/kenrowland/roxiebeat/vendor/github.com/elastic/beats/dev-tools/mage/target/build.CrossBuildGoDaemon
Running dependency: github.com/kenrowland/roxiebeat/vendor/github.com/elastic/beats/dev-tools/mage/target/build.CrossBuild
exec: docker info -f {{ json .}}
Running dependency: github.com/kenrowland/roxiebeat/vendor/github.com/elastic/beats/dev-tools/mage.buildMage
exec: mage -f -goos=linux -goarch=amd64 -compile build/mage-linux-amd64
crossBuild: Platform list = [{linux/386 cgo|xbuild|default} {linux/amd64 cgo|xbuild|default}]
Max parallel jobs = 0
crossBuild: Platform list = [{linux/386 xbuild|default|cgo} {linux/amd64 cgo|xbuild|default}]
Okay, now we're making progress.
This immediately looked suspicious:
So I forced that to be zero and now I can reproduce the issue.
Now, we need to figure out why that's zero. If we look at the code, the parallel job count is based on one of three things:
MAX_PARALLEL env varAre you running this in some kind of odd environment? If docker is installed from the host you're running this on, can you grab the value of docker info | grep -i cpu. I don't know what platform you're on, so can you make a quick go script or something to get the value of runtime.NumCPU()?
Alex,
I think I may have found something related to your last response.
In ../dev-tools/mage/common.go around line 445 (or so) is where the maxParallel value is set. it first sets it based on runtime.NumCPU(). This works fine and the value is 8. The GetDockerInfo however does not work because docker seems to want to run with root privileges. I figured this out, I believe, by commenting out the code that calls DockerInfo and instead use the 8 returned by the NumCPU() call. Here are those results:
mage -v package
... some removed to meet character limit ...
make: [kibana] Error 1 (ignored)
2019/09/11 16:25:09 Found Elastic Beats dir at /home/rowlke01/go/src/github.com/kenrowland/roxiebeat/vendor/github.com/elastic/beats
Running target: Config
>> Building roxiebeat.yml for linux/amd64
>> Building roxiebeat.reference.yml for linux/amd64
>> Building roxiebeat.docker.yml for linux/amd64
Updating generated files for roxiebeat
mkdir -p include
go run ./vendor/github.com/elastic/beats/dev-tools/cmd/asset/asset.go -license ASL2 -pkg include -in fields.yml -out include/fields.go roxiebeat
if [ -d /home/rowlke01/go/src/github.com/kenrowland/roxiebeat/_meta/kibana ]; then \
cp -pr /home/rowlke01/go/src/github.com/kenrowland/roxiebeat/_meta/kibana/* /home/rowlke01/go/src/github.com/kenrowland/roxiebeat/_meta/kibana.generated ; \
fi
Running dependency: github.com/kenrowland/roxiebeat/vendor/github.com/elastic/beats/dev-tools/mage/target/build.CrossBuildGoDaemon
Running dependency: github.com/kenrowland/roxiebeat/vendor/github.com/elastic/beats/dev-tools/mage/target/build.CrossBuild
exec: docker info -f {{ json .}}
Running dependency: github.com/kenrowland/roxiebeat/vendor/github.com/elastic/beats/dev-tools/mage.buildMage
exec: mage -f -goos=linux -goarch=amd64 -compile build/mage-linux-amd64
crossBuild: Platform list = [{linux/386 xbuild|default|cgo} {linux/amd64 default|cgo|xbuild}]
crossBuild: Platform list = [{linux/386 xbuild|default|cgo} {linux/amd64 default|cgo|xbuild}]
Max parallel jobs = 8
Parallel job waited 153.419µs before starting.
>> buildGoDaemon: Building for linux/amd64
Parallel job waited 2.85µs before starting.
Parallel job waited 544ns before starting.
>> buildGoDaemon: Building for linux/386
Parallel job waited 1.919µs before starting.
>> golangCrossBuild: Building for linux/amd64
exec: docker run --env EXEC_UID=1000 --env EXEC_GID=1000 --rm --env MAGEFILE_VERBOSE=true --env MAGEFILE_TIMEOUT= -v /home/rowlke01/go/src/github.com/kenrowland/roxiebeat:/go/src/github.com/kenrowland/roxiebeat -w /go/src/github.com/kenrowland/roxiebeat docker.elastic.co/beats-dev/golang-crossbuild:1.12.9-main-debian7 --build-cmd build/mage-linux-amd64 buildGoDaemon -p linux/amd64
exec: docker run --env EXEC_UID=1000 --env EXEC_GID=1000 --rm --env MAGEFILE_VERBOSE=true --env MAGEFILE_TIMEOUT= -v /home/rowlke01/go/src/github.com/kenrowland/roxiebeat:/go/src/github.com/kenrowland/roxiebeat -w /go/src/github.com/kenrowland/roxiebeat docker.elastic.co/beats-dev/golang-crossbuild:1.12.9-main-debian7 --build-cmd build/mage-linux-amd64 buildGoDaemon -p linux/386
>> golangCrossBuild: Building for linux/386
exec: docker run --env EXEC_UID=1000 --env EXEC_GID=1000 --rm --env MAGEFILE_VERBOSE=true --env MAGEFILE_TIMEOUT= -v /home/rowlke01/go/src/github.com/kenrowland/roxiebeat:/go/src/github.com/kenrowland/roxiebeat -w /go/src/github.com/kenrowland/roxiebeat docker.elastic.co/beats-dev/golang-crossbuild:1.12.9-main-debian7 --build-cmd build/mage-linux-amd64 golangCrossBuild -p linux/amd64
exec: docker run --env EXEC_UID=1000 --env EXEC_GID=1000 --rm --env MAGEFILE_VERBOSE=true --env MAGEFILE_TIMEOUT= -v /home/rowlke01/go/src/github.com/kenrowland/roxiebeat:/go/src/github.com/kenrowland/roxiebeat -w /go/src/github.com/kenrowland/roxiebeat docker.elastic.co/beats-dev/golang-crossbuild:1.12.9-main-debian7 --build-cmd build/mage-linux-amd64 golangCrossBuild -p linux/386
docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.40/containers/create: dial unix /var/run/docker.sock: connect: permission denied.
See 'docker run --help'.
docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.40/containers/create: dial unix /var/run/docker.sock: connect: permission denied.
See 'docker run --help'.
docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.40/containers/create: dial unix /var/run/docker.sock: connect: permission denied.
See 'docker run --help'.
docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.40/containers/create: dial unix /var/run/docker.sock: connect: permission denied.
See 'docker run --help'.
package ran for 29.494787632s
Error: running "docker run --env EXEC_UID=1000 --env EXEC_GID=1000 --rm --env MAGEFILE_VERBOSE=true --env MAGEFILE_TIMEOUT= -v /home/rowlke01/go/src/github.com/kenrowland/roxiebeat:/go/src/github.com/kenrowland/roxiebeat -w /go/src/github.com/kenrowland/roxiebeat docker.elastic.co/beats-dev/golang-crossbuild:1.12.9-main-debian7 --build-cmd build/mage-linux-amd64 golangCrossBuild -p linux/amd64" failed with exit code 126
running "docker run --env EXEC_UID=1000 --env EXEC_GID=1000 --rm --env MAGEFILE_VERBOSE=true --env MAGEFILE_TIMEOUT= -v /home/rowlke01/go/src/github.com/kenrowland/roxiebeat:/go/src/github.com/kenrowland/roxiebeat -w /go/src/github.com/kenrowland/roxiebeat docker.elastic.co/beats-dev/golang-crossbuild:1.12.9-main-debian7 --build-cmd build/mage-linux-amd64 golangCrossBuild -p linux/386" failed with exit code 126
running "docker run --env EXEC_UID=1000 --env EXEC_GID=1000 --rm --env MAGEFILE_VERBOSE=true --env MAGEFILE_TIMEOUT= -v /home/rowlke01/go/src/github.com/kenrowland/roxiebeat:/go/src/github.com/kenrowland/roxiebeat -w /go/src/github.com/kenrowland/roxiebeat docker.elastic.co/beats-dev/golang-crossbuild:1.12.9-main-debian7 --build-cmd build/mage-linux-amd64 buildGoDaemon -p linux/386" failed with exit code 126
running "docker run --env EXEC_UID=1000 --env EXEC_GID=1000 --rm --env MAGEFILE_VERBOSE=true --env MAGEFILE_TIMEOUT= -v /home/rowlke01/go/src/github.com/kenrowland/roxiebeat:/go/src/github.com/kenrowland/roxiebeat -w /go/src/github.com/kenrowland/roxiebeat docker.elastic.co/beats-dev/golang-crossbuild:1.12.9-main-debian7 --build-cmd build/mage-linux-amd64 buildGoDaemon -p linux/amd64" failed with exit code 126
There are a lot of permission denied responses when attempting to run the docker command.
Yep, the package process requires docker. You'll have to set up your environment such that the build system can start docker containers itself.
The numParallel() function should fall back to the go runtime CPU count if it can't access docker. This makes me wonder if we're somehow using the invalid value from docker info anyway.
Alex,
I found the solution. I am running on Ubuntu and in order to execute docker commands, for which we use sudo usually, as a non-root user, you have to add your user to the docker group.
Once I created the group and added myself to the group (and rebooted to pick up the change), the 'mage package' runs w/o any issues now.
Thanks for your help in getting the extra debug info that help lead me to the solution.
Ken
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.