Unable to GenerateCustomBeat with Git Bash on Windows

Mint-M · January 22, 2020, 3:05am

I have followed all steps of the official guide bellow to generate a custom beat.
https://www.elastic.co/guide/en/beats/devguide/current/newbeat-generate.html#newbeat-generate

It works on CentOS8 ,however I met a few problems on Windows.

Git 2.25.0 ,Python 2.7.17 ,go 1.13.5 and virtualenv 16.7.9 were installed on this Windows laptop.

I fixed the first problem of the wrong directory format myself by editing beats\generator\common\beatgen\setup\setup.go.

Line 100: "git archive --remote "+beatPath+" HEAD | tar -x --exclude=x-pack -C "+vendorPath) should be replaced with "git archive --remote '"+beatPath+"' HEAD | tar -x --exclude=x-pack -C "+vendorPath).

$ mage GenerateCustomBeat
2020/01/22 10:10:04 Found Elastic Beats dir at D:\go\src\github.com\elastic\beats
Enter the beat name [examplebeat]:
Enter your github name [your-github-name]:
Enter the beat path [github.com/your-github-name/examplebeat]:
Enter your full name [Firstname Lastname]:
Enter the beat type [beat]:
fatal: sent error to the client: git upload-archive: archiver died with error
remote: fatal: 'D:gosrcgithub.comelasticbeats' does not appear to be a git repository
remote: git upload-archive: archiver died with error
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors
Error: error running git archive: running "sh -c git archive --remote D:\go\src\github.com\elastic\beats HEAD | tar -x --exclude=x-pack -C vendor\github.com\elastic\beats" failed with exit code 2

Nevertheless, it caused some loops after running 'mage GenerateCustomBeat' again.
The output kept showing "Found Elastic Beats dir at C:\Users\beats\go\src\github.com\your-github-name\examplebeat\vendor\github.com\elastic\beats".
Unfortunately, I didn't find any useful message which could help resolve this problem.
It can also be reproduced on a fresh installed Windows computer.

beats@DESKTOP-T5E96U8 MINGW64 ~/go/src/github.com/elastic/beats (master)
$ mage -debug GenerateCustomBeat
DEBUG: 10:30:28.377736 getting all non-mage files in .
DEBUG: 10:30:28.555476 found non-mage files
DEBUG: 10:30:28.555476 getting all files plus mage files
DEBUG: 10:30:28.886081 time to scan for Magefiles: 508.3452ms
DEBUG: 10:30:28.886081 found magefiles: magefile.go
DEBUG: 10:30:29.033354 output exe is C:\Users\beats\magefile\e3fea17945c6cebfb332d4aa149fcb540de3c7f3.exe
DEBUG: 10:30:29.164922 go build cache exists, will ignore any compiled binary
DEBUG: 10:30:29.164922 parsing files
DEBUG: 10:30:29.166134 found target AddLicenseHeaders
DEBUG: 10:30:29.166134 found target CheckLicenseHeaders
DEBUG: 10:30:29.166134 found target DumpVariables
DEBUG: 10:30:29.166134 found target Fmt
DEBUG: 10:30:29.166134 found target GenerateCustomBeat
DEBUG: 10:30:29.166134 found target PackageBeatDashboards
DEBUG: 10:30:29.166134 time parse Magefiles: 1.1523ms
DEBUG: 10:30:29.166241 Creating mainfile at mage_output_file.go
DEBUG: 10:30:29.166442 writing new file at mage_output_file.go
DEBUG: 10:30:29.169910 compiling to C:\Users\beats\magefile\e3fea17945c6cebfb332d4aa149fcb540de3c7f3.exe
DEBUG: 10:30:29.169910 compiling using gocmd: go
DEBUG: 10:30:29.517677 running go build -o C:\Users\beats\magefile\e3fea17945c6cebfb332d4aa149fcb540de3c7f3.exe magefile.go mage_output_file.go
DEBUG: 10:30:30.660657 time to compile Magefile: 1.1290491s
DEBUG: 10:30:30.660804 running binary C:\Users\beats\magefile\e3fea17945c6cebfb332d4aa149fcb540de3c7f3.exe
DEBUG: 10:30:30.660804 running magefile with mage vars:
MAGEFILE_DEBUG=1
MAGEFILE_GOCMD=go
2020/01/22 10:30:30 Found Elastic Beats dir at C:\Users\beats\go\src\github.com\elastic\beats
Enter the beat name [examplebeat]:
Enter your github name [your-github-name]:
Enter the beat path [github.com/your-github-name/examplebeat]:
Enter your full name [Firstname Lastname]:
Enter the beat type [beat]:
You have uncommited changes in elastic/beats. Running CopyVendor running in dev mode, elastic/beats will be copied into the vendor directory with cp
DEBUG: 10:31:05.434840 getting all non-mage files in .
DEBUG: 10:31:06.824772 found non-mage files main.go
DEBUG: 10:31:06.824772 marked file as non-mage: "main.go"
DEBUG: 10:31:06.824772 getting all files plus mage files
DEBUG: 10:31:07.556151 time to scan for Magefiles: 2.1213114s
DEBUG: 10:31:07.556151 found magefiles: magefile.go
DEBUG: 10:31:07.708800 output exe is C:\Users\beats\magefile\7f339464e19330127d1e4d10c963fc90c91870d6.exe
DEBUG: 10:31:07.858847 go build cache exists, will ignore any compiled binary
DEBUG: 10:31:07.858847 parsing files
DEBUG: 10:31:07.858847 found target Build
DEBUG: 10:31:07.858847 found target BuildGoDaemon
DEBUG: 10:31:07.858847 found target Check
DEBUG: 10:31:07.858847 found target Clean
DEBUG: 10:31:07.858847 found target Config
DEBUG: 10:31:07.858847 found target CrossBuild
DEBUG: 10:31:07.858847 found target Fields
DEBUG: 10:31:07.858847 found target Fmt
DEBUG: 10:31:07.858847 found target GolangCrossBuild
DEBUG: 10:31:07.858847 found target Package
DEBUG: 10:31:07.858847 found target Test
DEBUG: 10:31:07.858847 found target Update
DEBUG: 10:31:07.858847 found target VendorUpdate
DEBUG: 10:31:07.858847 time parse Magefiles: 0s
DEBUG: 10:31:07.858847 Creating mainfile at mage_output_file.go
DEBUG: 10:31:07.859787 writing new file at mage_output_file.go
DEBUG: 10:31:07.862983 compiling to C:\Users\beats\magefile\7f339464e19330127d1e4d10c963fc90c91870d6.exe
DEBUG: 10:31:07.862983 compiling using gocmd: go
DEBUG: 10:31:08.232689 running go build -o C:\Users\beats\magefile\7f339464e19330127d1e4d10c963fc90c91870d6.exe magefile.go mage_output_file.go
DEBUG: 10:31:09.383308 time to compile Magefile: 1.1342371s
DEBUG: 10:31:09.383640 running binary C:\Users\beats\magefile\7f339464e19330127d1e4d10c963fc90c91870d6.exe
DEBUG: 10:31:09.383640 running magefile with mage vars:
MAGEFILE_DEBUG=1
MAGEFILE_GOCMD=go
MAGEFILE_DEBUG=1
MAGEFILE_GOCMD=go
2020/01/22 10:31:09 Found Elastic Beats dir at C:\Users\beats\go\src\github.com\your-github-name\examplebeat\vendor\github.com\elastic\beats
DEBUG: 10:31:09.849543 getting all non-mage files in .
DEBUG: 10:31:10.568216 found non-mage files main.go
DEBUG: 10:31:10.568216 marked file as non-mage: "main.go"
DEBUG: 10:31:10.568216 getting all files plus mage files
DEBUG: 10:31:11.293979 time to scan for Magefiles: 1.444436s
DEBUG: 10:31:11.293979 found magefiles: magefile.go
DEBUG: 10:31:11.461685 output exe is C:\Users\beats\magefile\7f339464e19330127d1e4d10c963fc90c91870d6.exe
DEBUG: 10:31:11.615653 go build cache exists, will ignore any compiled binary
DEBUG: 10:31:11.615653 parsing files
DEBUG: 10:31:11.615653 found target Build
DEBUG: 10:31:11.615653 found target BuildGoDaemon
DEBUG: 10:31:11.615653 found target Check
DEBUG: 10:31:11.615653 found target Clean
DEBUG: 10:31:11.615653 found target Config
DEBUG: 10:31:11.615653 found target CrossBuild
DEBUG: 10:31:11.615653 found target Fields
DEBUG: 10:31:11.615653 found target Fmt
DEBUG: 10:31:11.615653 found target GolangCrossBuild
DEBUG: 10:31:11.615653 found target Package
DEBUG: 10:31:11.615653 found target Test
DEBUG: 10:31:11.615653 found target Update
DEBUG: 10:31:11.615653 found target VendorUpdate
DEBUG: 10:31:11.615653 time parse Magefiles: 0s
DEBUG: 10:31:11.615653 Creating mainfile at mage_output_file.go
DEBUG: 10:31:11.616593 writing new file at mage_output_file.go
DEBUG: 10:31:11.618614 compiling to C:\Users\beats\magefile\7f339464e19330127d1e4d10c963fc90c91870d6.exe
DEBUG: 10:31:11.618614 compiling using gocmd: go
DEBUG: 10:31:11.959351 running go build -o C:\Users\beats\magefile\7f339464e19330127d1e4d10c963fc90c91870d6.exe magefile.go mage_output_file.go
DEBUG: 10:31:13.051396 time to compile Magefile: 1.0751484s
DEBUG: 10:31:13.051396 running binary C:\Users\beats\magefile\7f339464e19330127d1e4d10c963fc90c91870d6.exe
DEBUG: 10:31:13.051396 running magefile with mage vars:
MAGEFILE_DEBUG=1
MAGEFILE_GOCMD=go
MAGEFILE_DEBUG=1
MAGEFILE_GOCMD=go

jsoriano · January 23, 2020, 6:24pm

Hey @Mint-M, welcome to discuss

 remote: fatal: 'D:gosrcgithub.comelasticbeats' does not appear to be a git repository

This looks like a string where the backslashes are not being correctly escaped, it actually looks like a bug. Could you please open a new issue?

Thanks!

111191 · February 6, 2020, 4:25pm

Hi @jsoriano, we encountered the exact same issue.
Is there a bug opened?
In addition, is there a known work around we could use?

Mint-M · February 24, 2020, 11:23am

Thanks for your reply.
It can be fixed by editting 'beats\generator\common\beatgen\setup\setup.go' as I mentioned above. However,I was stucked at ' "Found Elastic Beats dir at C:\Users\beats\go\src[github.com](http://github.com/)\your-github-name\examplebeat\vendor[github.com](http://github.com/)\elastic\beats"' which kept showing after that fix.
The documentation of beats doesn't mention whether it is officially supported to generate custom beats on Beats and I haven't seen successful examples of generating beats on windows.
Is it possible for you to confirm if it can be reproduced by the elastic team and generating beats on windows will be supported?

system · March 23, 2020, 1:23pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
GenerateCustomBeat returns error for missing github.com/elastic/beats/v7 Beats beats-development	20	3315	June 2, 2020
Error creating new beat Beats beats-development	2	1136	May 9, 2020
Create new custom Beats, generate.py doesn't generate files/dir Beats beats-development	2	564	July 21, 2019
Beats Developer Guide [master] is out of date, creating new beat fails Beats beats-development	6	1066	March 8, 2021
`make setup` fails on new generated beat Beats	3	640	December 21, 2016

Unable to GenerateCustomBeat with Git Bash on Windows

Related topics