GenerateCustomBeat - logs was keep rooling with mage -debug GenerateCustomBeat

visasimbu · April 17, 2020, 10:49pm

I have followed below steps to create custom beat and which results me to infinite loop. I was digging for past 2 days. Kindly help me.

Go version source editor
go version go1.13.10 windows/amd64
Go environment variable

set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\myname\AppData\Local\go-build
set GOENV=C:\Users\myname\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=D:\MYDEV\ganesh\samplebeat\
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=c:\go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=c:\go\pkg\tool\windows_amd64
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\MYCOMP~1.SMO\AppData\Local\Temp\2\go-build195706982=/tmp/go-build -gno-record-gcc-switches

Create directory as per documentation

$ mkdir -p ${GOPATH}/src/github.com/elastic

Clone the custom beat code

$ git clone https://github.com/elastic/beats ${GOPATH}/src/github.com/elastic/beats
Cloning into 'D:\MYDEV\ganesh\samplebeat\/src/github.com/elastic/beats'...
remote: Enumerating objects: 4, done.
remote: Counting objects: 100% (4/4), done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 205668 (delta 0), reused 0 (delta 0), pack-reused 205664
Receiving objects: 100% (205668/205668), 232.73 MiB | 31.97 MiB/s, done.
Resolving deltas: 100% (124645/124645), done.
Checking out files: 100% (18274/18274), done.

Try generating custom beat

$ mage -debug GenerateCustomBeat
bash: mage: command not found

Due to mage file not found. Clone mage directory and set the mage exe to env variable.

myname@mymachine MINGW64 /d/MYDEV/ganesh/samplebeat
$ git clone https://github.com/magefile/mage
Cloning into 'mage'...
remote: Enumerating objects: 77, done.
remote: Counting objects: 100% (77/77), done.
remote: Compressing objects: 100% (62/62), done.
remote: Total 1981 (delta 28), reused 31 (delta 9), pack-reused 1904
Receiving objects: 100% (1981/1981), 7.79 MiB | 11.22 MiB/s, done.
Resolving deltas: 100% (969/969), done.
$ cd mage
$ go run bootstrap.go
Running target: Install
exec: go env GOBIN
exec: go env GOPATH
exec: git rev-parse --short HEAD
exec: git describe --tags
exec: go build -o D:\MYDEV\ganesh\samplebeat\bin\mage.exe -ldflags=-X "github.com/magefile/mage/mage.timestamp=2020-04-17T21:59:23Z" -X "github.com/magefile/mage/mage.commitHash=310e198" -X "github.com/magefile/mage/mage.gitTag=v1.9.0-12-g310e198" github.com/magefile/mage
myname@mymachine MINGW64 /d/MYDEV/ganesh/samplebeat/mage (master)
$ cd ..
myname@mymachine MINGW64 /d/MYDEV/ganesh/samplebeat
$ cd src/github.com/elastic/beats/

Try Create custom beat as per documentation

myname@mymachine MINGW64 /d/MYDEV/ganesh/samplebeat/src/github.com/elastic/beats (master)
$ mage -debug GenerateCustomBeat
DEBUG: 22:01:06.132134 getting all non-mage files in .
DEBUG: 22:01:08.451523 found non-mage files
DEBUG: 22:01:08.451523 getting all files plus mage files
DEBUG: 22:01:10.017316 time to scan for Magefiles: 3.8850816s
DEBUG: 22:01:10.017316 found magefiles: magefile.go
DEBUG: 22:01:10.044377 output exe is  C:\Users\myname\magefile\4f50871a7b9c2f8e90de6ea0a278a93206baca5d.exe
DEBUG: 22:01:10.075028 go build cache exists, will ignore any compiled binary
DEBUG: 22:01:10.075028 parsing files
DEBUG: 22:01:10.075028 found target AddLicenseHeaders
DEBUG: 22:01:10.075028 found target CheckLicenseHeaders
DEBUG: 22:01:10.075028 found target DumpVariables
DEBUG: 22:01:10.075028 found target Fmt
DEBUG: 22:01:10.075028 found target GenerateCustomBeat
DEBUG: 22:01:10.075028 found target PackageBeatDashboards
DEBUG: 22:01:10.075028 found target Vendor
DEBUG: 22:01:10.075028 time parse Magefiles: 0s
DEBUG: 22:01:10.075028 Creating mainfile at mage_output_file.go
DEBUG: 22:01:10.075028 writing new file at mage_output_file.go
DEBUG: 22:01:10.076021 compiling to C:\Users\myname\magefile\4f50871a7b9c2f8e90de6ea0a278a93206baca5d.exe
DEBUG: 22:01:10.077021 compiling using gocmd: go
DEBUG: 22:01:10.304650 running go build -o C:\Users\myname\magefile\4f50871a7b9c2f8e90de6ea0a278a93206baca5d.exe magefile.go mage_output_file.go
DEBUG: 22:01:12.261719 time to compile Magefile: 1.9520184s
DEBUG: 22:01:12.261719 running binary C:\Users\myname\magefile\4f50871a7b9c2f8e90de6ea0a278a93206baca5d.exe
DEBUG: 22:01:12.261719 running magefile with mage vars:
MAGEFILE_DEBUG=1
MAGEFILE_GOCMD=go
Enter the beat name [examplebeat]: test1
Enter your github name [your-github-name]: test2
Enter the beat path [github.com/test2/test1]: test3
Enter your full name [Firstname Lastname]: test4
Enter the beat type [beat]: beat
Enter the github.com/elastic/beats revision [master]: master
go: creating new go.mod: module test3
go: finding github.com master
...
...
go: extracting github.com/containerd/containerd v1.3.3
go: finding github.com/tsg/go-daemon v0.0.0-20200207173439-e704b93fd89b
DEBUG: 22:01:55.950169 getting all non-mage files in .
DEBUG: 22:01:56.858663 found non-mage files main.go
DEBUG: 22:01:56.858663 marked file as non-mage: "main.go"
DEBUG: 22:01:56.858663 getting all files plus mage files
DEBUG: 22:01:57.506874 time to scan for Magefiles: 1.5576661s
DEBUG: 22:01:57.506874 found magefiles: magefile.go
DEBUG: 22:01:57.534067 output exe is  C:\Users\myname\magefile\fa3afd537e6d4bf479585db0acb2048a7bbecfb0.exe
DEBUG: 22:01:57.565708 go build cache exists, will ignore any compiled binary
DEBUG: 22:01:57.565708 parsing files
DEBUG: 22:01:57.565708 found target Build
DEBUG: 22:01:57.565708 found target BuildGoDaemon
DEBUG: 22:01:57.565708 found target Check
DEBUG: 22:01:57.565708 found target Clean
DEBUG: 22:01:57.565708 found target Config
DEBUG: 22:01:57.565708 found target CrossBuild
DEBUG: 22:01:57.565708 found target Fields
DEBUG: 22:01:57.565708 found target Fmt
DEBUG: 22:01:57.565708 found target GolangCrossBuild
DEBUG: 22:01:57.565708 found target Package
DEBUG: 22:01:57.565708 found target Test
DEBUG: 22:01:57.565708 found target Update
DEBUG: 22:01:57.565708 found target VendorUpdate
DEBUG: 22:01:57.565708 time parse Magefiles: 0s
DEBUG: 22:01:57.565708 Creating mainfile at mage_output_file.go
DEBUG: 22:01:57.843391 writing new file at mage_output_file.go
DEBUG: 22:01:57.858458 compiling to C:\Users\myname\magefile\fa3afd537e6d4bf479585db0acb2048a7bbecfb0.exe
DEBUG: 22:01:57.858458 compiling using gocmd: go
DEBUG: 22:01:58.054503 running go build -o C:\Users\myname\magefile\fa3afd537e6d4bf479585db0acb2048a7bbecfb0.exe magefile.go mage_output_file.go
DEBUG: 22:02:00.647532 time to compile Magefile: 2.587994s
DEBUG: 22:02:00.647532 running binary C:\Users\myname\magefile\fa3afd537e6d4bf479585db0acb2048a7bbecfb0.exe
DEBUG: 22:02:00.647532 running magefile with mage vars:
MAGEFILE_VERBOSE=0
MAGEFILE_DEBUG=1
MAGEFILE_GOCMD=go
MAGEFILE_DEBUG=1
MAGEFILE_GOCMD=go
DEBUG: 22:02:00.821644 getting all non-mage files in .
DEBUG: 22:02:01.480696 found non-mage files main.go
DEBUG: 22:02:01.480696 marked file as non-mage: "main.go"
DEBUG: 22:02:01.480696 getting all files plus mage files
DEBUG: 22:02:02.110479 time to scan for Magefiles: 1.288802s
DEBUG: 22:02:02.110479 found magefiles: magefile.go
DEBUG: 22:02:02.138762 output exe is  C:\Users\myname\magefile\fa3afd537e6d4bf479585db0acb2048a7bbecfb0.exe
DEBUG: 22:02:02.169577 go build cache exists, will ignore any compiled binary
DEBUG: 22:02:02.169577 parsing files
DEBUG: 22:02:02.169577 found target Build
DEBUG: 22:02:02.169577 found target BuildGoDaemon
DEBUG: 22:02:02.169577 found target Check
DEBUG: 22:02:02.169577 found target Clean
DEBUG: 22:02:02.169577 found target Config
DEBUG: 22:02:02.169577 found target CrossBuild
DEBUG: 22:02:02.169577 found target Fields
DEBUG: 22:02:02.169577 found target Fmt
DEBUG: 22:02:02.169577 found target GolangCrossBuild
DEBUG: 22:02:02.169577 found target Package
DEBUG: 22:02:02.169577 found target Test
DEBUG: 22:02:02.169577 found target Update
DEBUG: 22:02:02.169577 found target VendorUpdate
DEBUG: 22:02:02.169577 time parse Magefiles: 0s
DEBUG: 22:02:02.169577 Creating mainfile at mage_output_file.go
DEBUG: 22:02:02.169577 writing new file at mage_output_file.go
DEBUG: 22:02:02.181575 compiling to C:\Users\myname\magefile\fa3afd537e6d4bf479585db0acb2048a7bbecfb0.exe
DEBUG: 22:02:02.181575 compiling using gocmd: go
DEBUG: 22:02:02.380083 running go build -o C:\Users\myname\magefile\fa3afd537e6d4bf479585db0acb2048a7bbecfb0.exe magefile.go mage_output_file.go
DEBUG: 22:02:03.253049 time to compile Magefile: 867.9426ms
DEBUG: 22:02:03.253049 running binary C:\Users\myname\magefile\fa3afd537e6d4bf479585db0acb2048a7bbecfb0.exe
DEBUG: 22:02:03.253049 running magefile with mage vars:
MAGEFILE_VERBOSE=0
MAGEFILE_DEBUG=1
MAGEFILE_GOCMD=go
MAGEFILE_DEBUG=1
MAGEFILE_GOCMD=go
**DEBUG: 22:02:03.336441 getting all non-mage files in .**
**DEBUG: 22:02:03.998261 found non-mage files main.go**
**DEBUG: 22:02:03.998261 marked file as non-mage: "main.go"**
**DEBUG: 22:02:03.998261 getting all files plus mage files**
**DEBUG: 22:02:04.621080 time to scan for Magefiles: 1.284606s**
**DEBUG: 22:02:04.621080 found magefiles: magefile.go**
**DEBUG: 22:02:04.648135 output exe is  C:\Users\myname\magefile\fa3afd537e6d4bf479585db0acb2048a7bbecfb0.exe**
**DEBUG: 22:02:04.678626 go build cache exists, will ignore any compiled binary**
**DEBUG: 22:02:04.678626 parsing files**
**DEBUG: 22:02:04.678626 found target Build**
**DEBUG: 22:02:04.678626 found target BuildGoDaemon**
**DEBUG: 22:02:04.678626 found target Check**
**DEBUG: 22:02:04.678626 found target Clean**
**DEBUG: 22:02:04.678626 found target Config**
**DEBUG: 22:02:04.678626 found target CrossBuild**
**DEBUG: 22:02:04.678626 found target Fields**
**DEBUG: 22:02:04.678626 found target Fmt**
**DEBUG: 22:02:04.678626 found target GolangCrossBuild**
**DEBUG: 22:02:04.678626 found target Package**
**DEBUG: 22:02:04.678626 found target Test**
**DEBUG: 22:02:04.678626 found target Update**
**DEBUG: 22:02:04.678626 found target VendorUpdate**
**DEBUG: 22:02:04.678626 time parse Magefiles: 0s**
**DEBUG: 22:02:04.678626 Creating mainfile at mage_output_file.go**
**DEBUG: 22:02:04.678626 writing new file at mage_output_file.go**
**DEBUG: 22:02:04.690626 compiling to C:\Users\myname\magefile\fa3afd537e6d4bf479585db0acb2048a7bbecfb0.exe**
**DEBUG: 22:02:04.690626 compiling using gocmd: go**
**DEBUG: 22:02:04.887450 running go build -o C:\Users\myname\magefile\fa3afd537e6d4bf479585db0acb2048a7bbecfb0.exe magefile.go mage_output_file.go**
**DEBUG: 22:02:05.770923 time to compile Magefile: 878.4584ms**
**DEBUG: 22:02:05.770923 running binary C:\Users\myname\magefile\fa3afd537e6d4bf479585db0acb2048a7bbecfb0.exe**
**DEBUG: 22:02:05.770923 running magefile with mage vars:**
**MAGEFILE_VERBOSE=0**
**MAGEFILE_DEBUG=1**
**MAGEFILE_GOCMD=go**
**MAGEFILE_DEBUG=1**
**MAGEFILE_GOCMD=go**
...

The above terminal is endless.. it was keep running with same blocks highlighted in bold. Kindly provide some insight

visasimbu · April 18, 2020, 3:55pm

After long wait. Received below error msg. Any help.

DEBUG: 02:56:02.752109 compiling using gocmd: go
DEBUG: 02:56:02.998797 running go build -o C:\Users\mycomp.myname\magefile\f5452d3fb0a833b59aaaabacd1664dcb18929b31.exe magefile.go mage_output_file.go
go tool compile: exit status 2
fatal error: out of memory allocating heap arena map

runtime stack:
runtime.throw(0xe73c9d, 0x27)
        c:/go/src/runtime/panic.go:774 +0x79 fp=0x23fa80 sp=0x23fa50 pc=0x431709
runtime.(*mheap).sysAlloc(0x15cc980, 0x2000, 0x7ff8c74c7200, 0x23fcd8)
        c:/go/src/runtime/malloc.go:711 +0x70a fp=0x23fb30 sp=0x23fa80 pc=0x40c03a
runtime.(*mheap).grow(0x15cc980, 0x1, 0x60001ffffffff)
        c:/go/src/runtime/mheap.go:1255 +0xaa fp=0x23fb80 sp=0x23fb30 pc=0x426bba
runtime.(*mheap).allocSpanLocked(0x15cc980, 0x1, 0x15f90a8, 0xbd993fff)
        c:/go/src/runtime/mheap.go:1170 +0x26d fp=0x23fc00 sp=0x23fb80 pc=0x426a3d
runtime.(*mheap).alloc_m(0x15cc980, 0x1, 0xac4e876c0023, 0x0)
        c:/go/src/runtime/mheap.go:1022 +0xd0 fp=0x23fc50 sp=0x23fc00 pc=0x4262a0
runtime.(*mheap).alloc.func1()
        c:/go/src/runtime/mheap.go:1093 +0x53 fp=0x23fc88 sp=0x23fc50 pc=0x45aae3
runtime.(*mheap).alloc(0x15cc980, 0x1, 0x10023, 0x7ff8c54f11c0)
        c:/go/src/runtime/mheap.go:1092 +0x91 fp=0x23fcd8 sp=0x23fc88 pc=0x426591
runtime.(*mcentral).grow(0x15cd7f0, 0x0)
        c:/go/src/runtime/mcentral.go:255 +0x82 fp=0x23fd18 sp=0x23fcd8 pc=0x417f22
runtime.(*mcentral).cacheSpan(0x15cd7f0, 0x23fdd8)
        c:/go/src/runtime/mcentral.go:106 +0x305 fp=0x23fd78 sp=0x23fd18 pc=0x417a45
runtime.(*mcache).refill(0x32f0008, 0x23)
        c:/go/src/runtime/mcache.go:138 +0x8c fp=0x23fd98 sp=0x23fd78 pc=0x4174cc
runtime.(*mcache).nextFree(0x32f0008, 0x23fe23, 0x4594cc, 0x4000, 0x0)
        c:/go/src/runtime/malloc.go:854 +0x8e fp=0x23fdd0 sp=0x23fd98 pc=0x40c34e
runtime.mallocgc(0x100, 0xdf2300, 0x32f4001, 0x4594f0)
        c:/go/src/runtime/malloc.go:1022 +0x7d2 fp=0x23fe70 sp=0x23fdd0 pc=0x40ccd2
runtime.newobject(0xdf2300, 0x15c4b60)
        c:/go/src/runtime/malloc.go:1151 +0x3f fp=0x23fea0 sp=0x23fe70 pc=0x40d0ef
runtime.mcommoninit(0x15c4b60)
        c:/go/src/runtime/proc.go:642 +0x126 fp=0x23fed8 sp=0x23fea0 pc=0x4344a6
runtime.schedinit()
        c:/go/src/runtime/proc.go:543 +0x80 fp=0x23ff30 sp=0x23fed8 pc=0x4340e0
runtime.rt0_go(0x23ff60, 0x7ff8c54f13d2, 0x23ff60, 0x0, 0x7ff8c54f13d2, 0x0, 0x0, 0x0, 0x0, 0x460a20, ...)
        c:/go/src/runtime/asm_amd64.s:214 +0x13d fp=0x23ff38 sp=0x23ff30 pc=0x45cf0d
DEBUG: 02:56:03.752830 time to compile Magefile: 749.0425ms
Error: error compiling magefiles
Error: running "make update" failed with exit code 1
Error: running "make update" failed with exit code 1
....

visasimbu · April 20, 2020, 8:39pm

Any insight.. kindly help me on this issue

system · May 15, 2020, 10:49pm

This topic was automatically closed after 28 days. New replies are no longer allowed.

Topic		Replies	Views
GenerateCustomBeats hangs Beats beats-development	1	476	March 18, 2021
Issues generating custom Beat Beats beats-development	1	782	December 23, 2021
GenerateCustomBeat Beats	2	459	March 24, 2020
Error creating new beat Beats beats-development	2	1137	May 9, 2020
Beats Developer Guide [master] is out of date, creating new beat fails Beats beats-development	6	1066	March 8, 2021

GenerateCustomBeat - logs was keep rooling with mage -debug GenerateCustomBeat

Related topics