Testing/Running this on a Windows 10 machine so that I can install it on a Dell EMC server.
I'm trying to utilize the Community Beat, Unitybeat, and I've installed Go and tested it out with the instruction from here, https://golang.org/doc/install?download=go1.14.6.windows-amd64.msi. The test installation that they have in these docs works perfectly fine and responds with hello world like its supposed to. Now I'm trying to create the binary for Unitybeat so I went to the github page (https://github.com/kckecheng/unitybeat), clicked on the green "Code" button and then downloaded the Zip file. I then extracted the zip file and placed it in the C:\Go\src folder which is where I put the hello world test installation. Then I ran .\make which is what the ReadMe says to do and I get this error
.\make
INFO: Could not find files for the given pattern(s). can't load package: package github.com/kckecheng/unitybeat/vendor/github.com/magefile/mage: cannot find package "." in: C:\Go\src\vendor\github.com\kckecheng\unitybeat\vendor\github.com\magefile\mage
'mage' is not recognized as an internal or external command,
operable program or batch file.
It seems like its a file path issue but I tried to change that in the main.go file but it didn't seem to work. I'm going to try and change the file path as that might be whats causing the issue but I could use some assistance on this. Thank you.
So I deleted everything and started over to try to pinpoint the underlying issues with this. I setup the Go installation following the tips from the YouTube video "How to Install Go on Windows 10" by ConceptClips so I have a C:\Go directory and a C:\gocode\ directory with bin, pkg and src folders in it. Here are the steps that I've done so far and Go is working. What I seem to have hit is a Mage installation or path issue, I'm not entirely sure what from the errors. Maybe I need to install Mage from the vendors folder?
- Deleted go with the instructions from here https://golang.org/doc/install#:~:text=Y.Z%20binary.-,Uninstalling%20Go,from%20your%20PATH%20environment%20variable
○ Removed the C:\Users\RDowney\go, C:\Go and Environment Variable, not sure why there were so many "go" folders
- Read this setup guide which says that we need the Go version 1.13.10 https://www.elastic.co/guide/en/beats/devguide/7.8/beats-contributing.html // I had installed version 1.14.6 or something originally which may have caused errors
- Go to the Go downloads page and find 1.13.10 and use the msi installer https://golang.org/dl/
○ This creates a folder in the C:\Go path that seems like itll run go from
○ Create a folder in the C:\ drive called gocode (C:\gocode)
○ Create three subfolders in the C:\gocode folder called bin, src and pkg
○ Open up the Windows Environment Variables and change the GOPATH to C:\gocode
○ Open up the PATH in the Windows Environment Variables and remove C:\Users\RDowney\go\bin and %USERPROFILE%\go\bin, then change to C:\gocode\bin
○ The GOPATH = C:\gocode
○ That’s what this section is telling you to do "After installing Go, set the GOPATH environment variable to point to your workspace location, and make sure $GOPATH/bin is in your PATH."
- Test the installation
○ Create a folder in C:\gocode\src and call it "hello"
○ Then add a file called hello.go in it with the information below
package main
import "fmt"
func main() {
fmt.Printf("hello, world\n")
}
○ Then build it with the Go tool by running go build hello.go in the C:\gocode\src\hello folder
○ Then type in hello to run the hello.exe file that the build just created, it should output hello, world and if that’s the case Go is working correctly
- Download the Elastic bests repository
○ Run cmd: cd .. // You should be in the C:\gocode\src\hello folder
○ Run cmd: mkdir %GOPATH%\src\github.com\elastic
○ Run cmd: dir // Check to see if it created the folders in the correct location
○ Run cmd: git clone https://github.com/elastic/beats %GOPATH%/src/github.com/elastic/beats
§ If you get an error, "fatal: unable to access : SSL certificate problem: unable to get local issuer certificate" try running git config --global http.sslVerify false // This allows you to get around the cert check
- Download the Unitybeat github repository
○ Run cmd: git clone https://github.com/kckecheng/unitybeat.git %GOPATH%/src/github.com/elastic/beats/unitybeat
○ Go to the unitybeat folder and run "make"
Running the command "make" produces this error. I skipped the make setup portion because it seems like that's for updating code etc. I'm just trying to produce the unitybeat.exe file.
C:\gocode\src\github.com\elastic\beats\kckecheng\unitybeat>make
INFO: Could not find files for the given pattern(s).
go: finding github.com/kckecheng/unitybeat/vendor/github.com/magefile/mage latest
go: finding github.com/kckecheng/unitybeat latest
go: github.com/kckecheng/unitybeat/vendor/github.com/magefile/mage: github.com/kckecheng/unitybeat/vendor/github.com/magefile/mage@v0.0.0-20190308053128-5d71b2541e27: parsing go.mod:
module declares its path as: github.com/magefile/mage
but was required as: github.com/kckecheng/unitybeat/vendor/github.com/magefile/mage
'mage' is not recognized as an internal or external command,
operable program or batch file.
The other day I was able to get around this error by working this GitHub thread (below) but that just created the next error that I had to troubleshoot and I couldnt get past that one. Honestly this seems like a file path issue but at this point I'm pretty lost.
If I change the Windows Environment variable to "MANGFILE_CACHE C:\gocode\src\github.com\elastic\beats\kckecheng\unitybeat\vendor\github.com\magefile\mage.mage", Mage will run but then throw the next error as I discussed.
C:\gocode\src\github.com\elastic\beats\kckecheng\unitybeat\vendor\github.com\magefile\mage>go run bootstrap.go
Running target: Install
exec: go env GOPATH
exec: git rev-parse --short HEAD
exec: git describe --tags
fatal: No names found, cannot describe anything.
exec: go build -o C:\gocode\bin\mage.exe -ldflags=-X "github.com/magefile/mage/mage.timestamp=2020-08-03T14:06:23-04:00" -X "github.com/magefile/mage/mage.commitHash=5d71b25" -X "github.com/magefile/mage/mage.gitTag=dev" github.com/magefile/mage
I updated the environment variables and path for Mage which seems to have moved this on to the next issue. When I run "make build" this is what happens.
C:\gocode\src\github.com\elastic\beats\kckecheng\unitybeat>make build
C:\gocode\bin\mage.exe
2020/08/03 16:04:36 Found Elastic Beats dir at C:\gocode\src\github.com\elastic\beats\kckecheng\unitybeat\vendor\github.com\elastic\beats
>> build: Building unitybeat
# github.com/elastic/beats/libbeat/common/kubernetes
C:\gocode\pkg\mod\github.com\elastic\beats@v7.6.2+incompatible\libbeat\common\kubernetes\informer.go:53:18: not enough arguments in call to p.List
have ("k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions)
want (context.Context, "k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions)
C:\gocode\pkg\mod\github.com\elastic\beats@v7.6.2+incompatible\libbeat\common\kubernetes\informer.go:57:19: not enough arguments in call to p.Watch
have ("k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions)
want (context.Context, "k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions)
C:\gocode\pkg\mod\github.com\elastic\beats@v7.6.2+incompatible\libbeat\common\kubernetes\informer.go:66:18: not enough arguments in call to e.List
have ("k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions)
want (context.Context, "k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions)
C:\gocode\pkg\mod\github.com\elastic\beats@v7.6.2+incompatible\libbeat\common\kubernetes\informer.go:69:19: not enough arguments in call to e.Watch
have ("k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions)
want (context.Context, "k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions)
C:\gocode\pkg\mod\github.com\elastic\beats@v7.6.2+incompatible\libbeat\common\kubernetes\informer.go:79:18: not enough arguments in call to n.List
have ("k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions)
want (context.Context, "k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions)
C:\gocode\pkg\mod\github.com\elastic\beats@v7.6.2+incompatible\libbeat\common\kubernetes\informer.go:83:19: not enough arguments in call to n.Watch
have ("k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions)
want (context.Context, "k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions)
C:\gocode\pkg\mod\github.com\elastic\beats@v7.6.2+incompatible\libbeat\common\kubernetes\informer.go:93:19: not enough arguments in call to ns.List
have ("k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions)
want (context.Context, "k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions)
C:\gocode\pkg\mod\github.com\elastic\beats@v7.6.2+incompatible\libbeat\common\kubernetes\informer.go:97:20: not enough arguments in call to ns.Watch
have ("k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions)
want (context.Context, "k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions)
C:\gocode\pkg\mod\github.com\elastic\beats@v7.6.2+incompatible\libbeat\common\kubernetes\informer.go:106:18: not enough arguments in call to d.List
have ("k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions)
want (context.Context, "k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions)
C:\gocode\pkg\mod\github.com\elastic\beats@v7.6.2+incompatible\libbeat\common\kubernetes\informer.go:109:19: not enough arguments in call to d.Watch
have ("k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions)
want (context.Context, "k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions)
C:\gocode\pkg\mod\github.com\elastic\beats@v7.6.2+incompatible\libbeat\common\kubernetes\informer.go:109:19: too many errors
Error: running "go build -o unitybeat.exe -ldflags -s -X github.com/elastic/beats/kckecheng/unitybeat/vendor/github.com/elastic/beats/libbeat/version.buildTime=2020-08-03T20:04:36Z -X github.com/elastic/beats/kckecheng/unitybeat/vendor/github.com/elastic/beats/libbeat/version.commit=5d71b2541e27a11b2d2ec7fcf8fd49eaf4344242" failed with exit code 2
I think the "make build" command was just a misunderstanding on my part of what the error message was indicating. I've been trying this out from two folders because I'm not exactly sure which file path is really the correct one to use or if both work fine. Each provides different errors unfortunately.
If I run "make" from C:\gocode\src\github.com\elastic\beats\kckecheng\unitybeat I get:
C:\gocode\src\github.com\elastic\beats\kckecheng\unitybeat>make
C:\gocode\bin\mage.exe
2020/08/04 10:41:42 Found Elastic Beats dir at C:\gocode\src\github.com\elastic\beats\kckecheng\unitybeat\vendor\github.com\elastic\beats
Targets:
build builds the Beat binary.
buildGoDaemon builds the go-daemon binary (use crossBuildGoDaemon).
clean cleans all generated files and build artifacts.
crossBuild cross-builds the beat for all target platforms.
crossBuildGoDaemon cross-builds the go-daemon binary using Docker.
fields generates a fields.yml for the Beat.
goTestIntegration executes the Go integration tests.
goTestUnit executes the Go unit tests.
golangCrossBuild build the Beat binary inside of the golang-builder.
package packages the Beat for distribution.
testPackages tests the generated packages (i.e.
update updates the generated files (aka make update).
And if I try it from the C:\gocode\src\github.com\kckecheng\unitybeat
C:\gocode\src\github.com\kckecheng\unitybeat>make
C:\gocode\bin\mage.exe
Targets:
build builds the Beat binary.
buildGoDaemon builds the go-daemon binary (use crossBuildGoDaemon).
clean cleans all generated files and build artifacts.
crossBuild cross-builds the beat for all target platforms.
crossBuildGoDaemon cross-builds the go-daemon binary using Docker.
fields generates a fields.yml for the Beat.
goTestIntegration executes the Go integration tests.
goTestUnit executes the Go unit tests.
golangCrossBuild build the Beat binary inside of the golang-builder.
package packages the Beat for distribution.
testPackages tests the generated packages (i.e.
update updates the generated files (aka make update).
If I run "mage build" from C:\gocode\src\github.com\elastic\beats\kckecheng\unitybeat:
C:\gocode\src\github.com\elastic\beats\kckecheng\unitybeat>mage build
2020/08/04 10:45:55 Found Elastic Beats dir at C:\gocode\src\github.com\elastic\beats\kckecheng\unitybeat\vendor\github.com\elastic\beats
>> build: Building unitybeat
# github.com/elastic/beats/libbeat/common/kubernetes
C:\gocode\pkg\mod\github.com\elastic\beats@v7.6.2+incompatible\libbeat\common\kubernetes\informer.go:53:18: not enough arguments in call to p.List
have ("k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions)
want (context.Context, "k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions)
C:\gocode\pkg\mod\github.com\elastic\beats@v7.6.2+incompatible\libbeat\common\kubernetes\informer.go:57:19: not enough arguments in call to p.Watch
have ("k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions)
want (context.Context, "k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions)
C:\gocode\pkg\mod\github.com\elastic\beats@v7.6.2+incompatible\libbeat\common\kubernetes\informer.go:66:18: not enough arguments in call to e.List
have ("k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions)
want (context.Context, "k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions)
C:\gocode\pkg\mod\github.com\elastic\beats@v7.6.2+incompatible\libbeat\common\kubernetes\informer.go:69:19: not enough arguments in call to e.Watch
have ("k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions)
want (context.Context, "k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions)
C:\gocode\pkg\mod\github.com\elastic\beats@v7.6.2+incompatible\libbeat\common\kubernetes\informer.go:79:18: not enough arguments in call to n.List
have ("k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions)
want (context.Context, "k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions)
C:\gocode\pkg\mod\github.com\elastic\beats@v7.6.2+incompatible\libbeat\common\kubernetes\informer.go:83:19: not enough arguments in call to n.Watch
have ("k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions)
want (context.Context, "k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions)
C:\gocode\pkg\mod\github.com\elastic\beats@v7.6.2+incompatible\libbeat\common\kubernetes\informer.go:93:19: not enough arguments in call to ns.List
have ("k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions)
want (context.Context, "k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions)
C:\gocode\pkg\mod\github.com\elastic\beats@v7.6.2+incompatible\libbeat\common\kubernetes\informer.go:97:20: not enough arguments in call to ns.Watch
have ("k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions)
want (context.Context, "k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions)
C:\gocode\pkg\mod\github.com\elastic\beats@v7.6.2+incompatible\libbeat\common\kubernetes\informer.go:106:18: not enough arguments in call to d.List
have ("k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions)
want (context.Context, "k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions)
C:\gocode\pkg\mod\github.com\elastic\beats@v7.6.2+incompatible\libbeat\common\kubernetes\informer.go:109:19: not enough arguments in call to d.Watch
have ("k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions)
want (context.Context, "k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions)
C:\gocode\pkg\mod\github.com\elastic\beats@v7.6.2+incompatible\libbeat\common\kubernetes\informer.go:109:19: too many errors
Error: running "go build -o unitybeat.exe -ldflags -s -X github.com/elastic/beats/kckecheng/unitybeat/vendor/github.com/elastic/beats/libbeat/version.buildTime=2020-08-04T14:45:55Z -X github.com/elastic/beats/kckecheng/unitybeat/vendor/github.com/elastic/beats/libbeat/version.commit=5d71b2541e27a11b2d2ec7fcf8fd49eaf4344242" failed with exit code 2
If I run "mage build" from C:\gocode\src\github.com\kckecheng\unitybeat:
C:\gocode\src\github.com\kckecheng\unitybeat>mage build
>> build: Building unitybeat
..\storagemetric\utils\log.go:7:2: cannot find package "github.com/sirupsen/logrus" in any of:
c:\go\src\github.com\sirupsen\logrus (from $GOROOT)
C:\gocode\src\github.com\sirupsen\logrus (from $GOPATH)
Error: running "go build -o unitybeat.exe -ldflags -X github.com/kckecheng/unitybeat/vendor/github.com/elastic/beats/libbeat/version.buildTime=2020-08-04T14:46:20Z -X github.com/kckecheng/unitybeat/vendor/github.com/elastic/beats/libbeat/version.commit=5d71b2541e27a11b2d2ec7fcf8fd49eaf4344242" failed with exit code 1
I'm not entirely sure if this is a windows thing or not since i work on a mac, but a couple of things to note:
unitybeat seems to be trying to copy around the beats repo that's checked out in the gopath in order to vendor it when you run make setup per their README
it doesn't appear to be adhering to any sort of version pinning
the master branch for beats likely has diverged quite a bit from whatever unitybeat was originally built for
That said, it looks like the README for that project isn't accurate--they already have a copied version of beats in their repo. I was able to build the project running the following;
➜ unitybeat git:(master) ✗ make -f ./vendor/github.com/elastic/beats/libbeat/scripts/Makefile mage # this installs mage
➜ unitybeat git:(master) ✗ go get github.com/kckecheng/storagemetric/dell/emc/unity # this installs the library dependency that the project requires
➜ unitybeat git:(master) ✗ mage build # this builds the beat
@andrewstucki is correct. The supported way of generating Beats has changed a lot in the last few months as we adopted Golang 1.14. Unfortunately, the author of this Beat has not followed up the changes.
Let me know if you still have problems after trying the suggestions above. We could try move the Beat to go modules properly, so the Beat can use the latest Beats framework.
Thank you @andrewstucki and @kvch for taking the time to look at this. It seems like with a few of these issues I really would have just been going around and around trying to get this to work. I will start work on this now and will hopefully have a response for both of you by EOD today (4pm EST) on my status. Hopefully a lot quicker.
***Update I found some info on Stack about the gcc error so I installed it and set the path. Still seeing the same error.
From the looks of the last command Mage is executing and trying to build Unitybeat but I've hit this error. Seems like I either need gcc or add the gcc executable file to the %PATH% which is easy enough if I knew where that was. Doing some research now.
C:\gocode\src\github.com\kckecheng\unitybeat>make build
C:\gocode\bin\mage.exe
>> build: Building unitybeat
# github.com/kckecheng/unitybeat/vendor/github.com/elastic/beats/vendor/github.com/DataDog/zstd
exec: "gcc": executable file not found in %PATH%
Error: running "go build -o unitybeat.exe -ldflags -X github.com/kckecheng/unitybeat/vendor/github.com/elastic/beats/libbeat/version.buildTime=2020-08-05T16:44:50Z -X github.com/kckecheng/unitybeat/vendor/github.com/elastic/beats/libbeat/version.commit=5d71b2541e27a11b2d2ec7fcf8fd49eaf4344242" failed with exit code 2
As a follow up to this, if I build this using a Linux server can I just copy the Unitybeat folder and utilize it on a Windows machine? I feel like the obvious answer is no but I've got to ask if it could save you all time. From the quick research I've done the Dell EMC servers are Windows based so if you say its necessary to build via Windows then thats what I'll have to do.
*** 10:09 PM EST and I finally got this to work. What a miracle this was lol. I had to start from complete scratch on my home Windows computer and follow each each message until no more were showing. What a mess. ***
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
