Make package: yaml.v2': No such file or directory

Elastic Stack Beats
1

Hello,
I am getting the following error when i try to package a newly generated beat (with cookiecutter) on debian jessie:

$ make package
Cloning into '/go/src/github.com/tsg/gotpl'...
cp: cannot stat '/go/src/github.com/elastic/beats/vendor/gopkg.in/yaml.v2': No such file or directory
vendor/github.com/elastic/beats/libbeat/scripts/Makefile:335: recipe for target 'prepare-package' failed
make[1]: *** [prepare-package] Error 1
make[1]: Leaving directory '/go/src/github.com/user/test'
vendor/github.com/elastic/beats/libbeat/scripts/Makefile:388: recipe for target 'package' failed
make: *** [package] Error 2

$ uname -ra
Linux dev1 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u3 (2016-01-17) x86_64 GNU/Linux
$ echo $GOPATH
/home/user/go_workspace
$ go version
go version go1.7.3 linux/amd64
$ python --version
Python 2.7.9

To reproduce:
$ mkdir $GOPATH/src/github.com/{user}
$ cd $GOPATH/src/github.com/{user}
$ cookiecutter $GOPATH/src/github.com/elastic/beats/generate/beat
project_name [Examplebeat]: test
github_name [your-github-name]: user
beat [countbeat]:
beat_path [github.com/{github id}]:
full_name [Firstname Lastname]: {Full Name}
$ cd $GOPATH/src/github.com/{user}/test
$ make setup
$ make package-setup
$ make package

I have also tried same steps under the roor user and with GOPATH=/go. This results in the same error.

2

Can you check if gopkg.in/yaml.v2 exists in your vendored copy of the beats project? In your case that would mean in: vendor/github.com/elastic/beats/vendor/gopkg.in/yaml.v2

With the most recent version of the generator you should only have to run make and then make package. No need for the package-setup part anymore.

The GOPATH is different from your gopath, as all this is running inside docker containers.

3

Additional comment: What is the version of your beats repository? Can you make sure to update it to master?

4

Thank you for a quick reply. The following folder exists with needed content:

$GOPATH/src/github.com/user/test18112016/vendor/github.com/elastic/beats/vendor/gopkg.in/yaml.v2/

I was also able to reproduce the issue today one more time just several minutes ago after clearing the $GOPATH completely (which i hope mean that i got the latest master version for beats ):

rm -rf $GOPATH/*
go get github.com/elastic/beats
mkdir $GOPATH/src/github.com/user
cd $GOPATH/src/github.com/user
cookiecutter $GOPATH/src/github.com/elastic/beats/generate/beat
project_name [Examplebeat]: test18112016
github_name [your-github-name]: user
beat [test18112016]:
beat_path [github.com/{github id}]:
full_name [Firstname Lastname]: 
cd $GOPATH/src/github.com/user/test18112016
make setup
make
make package

This resulted in the same error:

Cloning into '/go/src/github.com/tsg/gotpl'...
cp: cannot stat '/go/src/github.com/elastic/beats/vendor/gopkg.in/yaml.v2': No such file or directory

The

/go/...

root dirictory is not available and as far i understand it should not look for it at all due to my

$GOPATH=/home/user/go_workspace

Thank you in advance.

5

The part i miss above is go get github.com/elastic/beats/libbeat. Do you run that after you removed all the content of the go directory? If not, I'm kind of confused why cookiecutter would still work?

6

Sorry, the following step was also done:

go get github.com/elastic/beats

As per documentation:

https://www.elastic.co/guide/en/beats/libbeat/current/new-beat.html

I updated my previous reply.

7

Can you update to Golang 1.7? I think it should also work under 1.6 but just to be sure we are testing the same version.

8

Seems like I already have the latest go version:

$ go version
go version go1.7.3 linux/amd64

Are you able to reproduce the issue on your side?

9

Strangely i can't but I'm working on OS X. Can you check if yaml.v2 exists in your vendor beats directory?

10

Yes, i see it is available:

user@dev1:~/go_workspace/src/github.com/user/test22112016$ find /home/user/go_workspace/ -name yaml.v2
/home/user/go_workspace/src/gopkg.in/yaml.v2
/home/user/go_workspace/src/github.com/elastic/beats/vendor/gopkg.in/yaml.v2
/home/user/go_workspace/src/github.com/user/test22112016/vendor/github.com/elastic/beats/vendor/gopkg.in/yaml.v2

And the error is:

Compiling import_dashboards for darwin/amd64
Cloning into '/go/src/github.com/tsg/gotpl'...
cp: cannot stat '/go/src/github.com/elastic/beats/vendor/gopkg.in/yaml.v2': No such file or directory
vendor/github.com/elastic/beats/libbeat/scripts/Makefile:335: recipe for target 'prepare-package' failed
make[1]: *** [prepare-package] Error 1
11

That is quite strange as all this happens inside the docker container. Which version of docker and docker-compose are you using?

As far as I understand running make to build the binary works?

12

Yep the

make

command does not return any errors. Docker version is:

user@dev1:~$ docker --version
Docker version 1.12.3, build 6b644ec

I don't have the docker-compose installed. Should it be?

13

That is quite strange. I assume you have yaml.v2 only in the vendor directory of beats and not in the global GOPATH? Docker version also looks good. You are right that compose is not needed for the packaging part.

My tests are done on Mac OS Sierra. I'm getting the impression there is something going wrong with mounting the volumes into the docker container. As far as I can see you are on Debian?

14

I have some unknown yaml.v2.a file, not directory, under the gopath:

user@dev1:~$ $echo $GOPATH
    -bash: /home/user/go_workspace: Is a directory
user@dev1:~$ ls -ltr /home/user/go_workspace/pkg/linux_amd64/gopkg.in/yaml.v2.a
    -rw-r--r-- 1 user user 513782 Nov 22 15:29 /home/user/go_workspace/pkg/linux_amd64/gopkg.in/yaml.v2.a

Yep, trying to make it work on debian:

~$ user@dev1:~$ lsb_release -da
    No LSB modules are available.
    Distributor ID: Debian
    Description:    Debian GNU/Linux 8.3 (jessie)
    Release:        8.3
    Codename:       jessie
15

Sorry for the late reply. Is what you shared above the full output after make package or are there more lines? I would expect quite a few additional lines before.

16

I think I found the problem and here is the potential fix for it: https://github.com/elastic/beats/pull/3094

I was looking in the completely wrong place initially as I thought it happens directly after make package, but there are quite a lot of log lines in between. Not sure yet why it worked actually in the past :slight_smile:

17

Please let me know if you need me to reproduce the issue to get the make package output.

18

@Aliaksandr_Zabrodski It would be nice if you could keep an eye on the PR and retry as soon as it is merged and let me know if it fixes the issue for you.

19

Hi @ruflin,

I tried the PR and the problem is gone, but soon afterwards I run into another problem that the beat template file is not found, in my particular case I get the following message:

cp: cannot stat 'httpbeat.template.json': No such file or directory.

20

Please ignore the previous comment, the files where not in the correct place.
So I think the PR works as expected