Make package: yaml.v2': No such file or directory

Aliaksandr_Zabrodski · December 27, 2016, 3:52pm

I have tried the fix today and got another similar error at make package stage:

echo Creating DEB packages for test27122016
Creating DEB packages for test27122016
ARCH=386 BEAT=test27122016 BUILD_DIR=/home/user/go_workspace/src/github.com/user/test27122016/build BEAT_DIR=/home/user/go_workspace/src/github.com/user/test27122016 BUILDID= SNAPSHOT=yes /home/user/go_workspace/src/github.com/user/test27122016/vendor/github.com/elastic/beats/dev-tools/packer/platforms/debian/build.sh
/home/user/go_workspace/src/github.com/user/test27122016/vendor/github.com/elastic/beats/dev-tools/packer/platforms/debian/build.sh: 12: /home/user/go_workspace/src/github.com/user/test27122016/vendor/github.com/elastic/beats/dev-tools/packer/platforms/debian/build.sh: gotpl: not found
Makefile:13: recipe for target 'test27122016/deb' failed
make[1]: *** [test27122016/deb] Error 127
make[1]: Leaving directory '/home/user/go_workspace/src/github.com/user/test27122016/vendor/github.com/elastic/beats/dev-tools/packer'
vendor/github.com/elastic/beats/libbeat/scripts/Makefile:400: recipe for target 'package' failed
make: *** [package] Error 2

Extended make package output is below:

...    
Fetching dependencies...
Execute /scripts/before_build.sh github.com/user/test27122016
install -d -m 755 /build/homedir/scripts/
install -m 755 ./vendor/github.com/elastic/beats/libbeat/scripts/migrate_beat_config_1_x_to_5_0.py /build/homedir/scripts/
Compiling import_dashboards for linux/amd64
Compiling import_dashboards for linux/386
Compiling import_dashboards for windows/amd64
Compiling import_dashboards for windows/386
Compiling import_dashboards for darwin/amd64
Cloning into '/go/src/github.com/tsg/gotpl'...
make: Nothing to be done for `before-build'.
Compiling test27122016 for linux/amd64...
Compiling test27122016 for linux/386...
Compiling test27122016 for windows-4.0/amd64...
Compiling test27122016 for windows-4.0/386...
Compiling test27122016 for darwin/amd64...
Build process completed
make[1]: Leaving directory '/home/user/go_workspace/src/github.com/user/test27122016'
SNAPSHOT=yes BUILDID= BEAT_DIR=github.com/user/test27122016 BUILD_DIR=/home/user/go_workspace/src/github.com/user/test27122016/build make -C ./vendor/github.com/elastic/beats/dev-tools/packer test27122016/deb test27122016/rpm test27122016/darwin test27122016/win test27122016/bin /home/user/go_workspace/src/github.com/user/test27122016/build/upload/build_id.txt
make[1]: Entering directory '/home/user/go_workspace/src/github.com/user/test27122016/vendor/github.com/elastic/beats/dev-tools/packer'
docker run --rm -v /home/user/go_workspace/src/github.com/user/test27122016/build:/build tudorg/go-daemon
Fetching go-daemon
Cloning into 'go-daemon'...
Compiling for linux/amd64..
/tmp/ccHpxljV.o: In function `main':
god.c:(.text+0x2ca): warning: Using 'getgrnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
god.c:(.text+0x262): warning: Using 'getpwnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
Compiling for linux/i386..
/tmp/ccIl6uos.o: In function `main':
god.c:(.text+0x2ee): warning: Using 'getgrnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
god.c:(.text+0x287): warning: Using 'getpwnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
Copying to host..
docker build --rm=true -t tudorg/fpm /home/user/go_workspace/src/github.com/user/test27122016/vendor/github.com/elastic/beats/dev-tools/packer/docker/fpm-image
Sending build context to Docker daemon 2.048 kB
Step 1 : FROM ubuntu:16.04
 ---> e4415b714b62
Step 2 : MAINTAINER Tudor Golubenco <tudor@elastic.co>
 ---> Using cache
 ---> 5563b321f868
Step 3 : RUN apt-get update &&     apt-get install -y --no-install-recommends         build-essential ruby-dev rpm zip dos2unix libgmp3-dev
 ---> Using cache
 ---> 0f14779ca09c
Step 4 : RUN gem install fpm
 ---> Using cache
 ---> 5def1a2c73a9
Successfully built 5def1a2c73a9
echo Creating DEB packages for test27122016
Creating DEB packages for test27122016
ARCH=386 BEAT=test27122016 BUILD_DIR=/home/user/go_workspace/src/github.com/user/test27122016/build BEAT_DIR=/home/user/go_workspace/src/github.com/user/test27122016 BUILDID= SNAPSHOT=yes /home/user/go_workspace/src/github.com/user/test27122016/vendor/github.com/elastic/beats/dev-tools/packer/platforms/debian/build.sh
/home/user/go_workspace/src/github.com/user/test27122016/vendor/github.com/elastic/beats/dev-tools/packer/platforms/debian/build.sh: 12: /home/user/go_workspace/src/github.com/user/test27122016/vendor/github.com/elastic/beats/dev-tools/packer/platforms/debian/build.sh: gotpl: not found
Makefile:13: recipe for target 'test27122016/deb' failed
make[1]: *** [test27122016/deb] Error 127
make[1]: Leaving directory '/home/user/go_workspace/src/github.com/user/test27122016/vendor/github.com/elastic/beats/dev-tools/packer'
vendor/github.com/elastic/beats/libbeat/scripts/Makefile:400: recipe for target 'package' failed
make: *** [package] Error 2

ruflin · January 3, 2017, 8:29am

We very recently change some code related to gotpl and these templates. Any chance you could again update to the most recent version of libbeat (master) and check again. Make sure to run make clean in advance.

Aliaksandr_Zabrodski · January 3, 2017, 9:03am

Hm, i got the same error:

echo "Start building packages for test03012017"
Start building packages for test03012017
mkdir -p /home/user/go_workspace/src/github.com/user/test03012017/build/upload

Generates the package.yml file with all information needed to create packages

...

make[1]: Entering directory '/home/user/go_workspace/src/github.com/user/test03012017'
fatal: Not a git repository (or any of the parent directories): .git

cross compile on ubuntu

docker run --rm
-v /home/user/go_workspace/src/github.com/user/test03012017/vendor/github.com/elastic/beats/dev-tools/packer/xgo-scripts:/scripts
-v /home/user/go_workspace/src/github.com/user/test03012017/vendor/github.com/elastic/beats/dev-tools/packer:/templates
-v /home/user/go_workspace/src/github.com/user:/source
-v /home/user/go_workspace/src/github.com/user/test03012017/build:/build
-e PUREGO="yes"
-e PACK=test03012017
-e BEFORE_BUILD=before_build.sh
-e SOURCE=/source
-e TARGETS="linux/amd64 linux/386 windows/amd64 windows/386 darwin/amd64"
-e BUILDID=
tudorg/beats-builder
github.com/user/test03012017
Working directory=/go/src/github.com/user
Copying main source folder /source to folder /go/src/github.com/user
Fetching dependencies...
Execute /scripts/before_build.sh github.com/user/test03012017
install -d -m 755 /build/homedir/scripts/
install -m 755 ./vendor/github.com/elastic/beats/libbeat/scripts/migrate_beat_config_1_x_to_5_0.py /build/homedir/scripts/
Compiling import_dashboards for linux/amd64
Compiling import_dashboards for linux/386
Compiling import_dashboards for windows/amd64
Compiling import_dashboards for windows/386
Compiling import_dashboards for darwin/amd64
Cloning into '/go/src/github.com/tsg/gotpl'...
make: Nothing to be done for before-build'. Compiling test03012017 for linux/amd64... Compiling test03012017 for linux/386... Compiling test03012017 for windows-4.0/amd64... Compiling test03012017 for windows-4.0/386... Compiling test03012017 for darwin/amd64... Build process completed make[1]: Leaving directory '/home/user/go_workspace/src/github.com/user/test03012017' SNAPSHOT=yes BUILDID= BEAT_DIR=github.com/user/test03012017 BUILD_DIR=/home/user/go_workspace/src/github.com/user/test03012017/build make -C ./vendor/github.com/elastic/beats/dev-tools/packer test03012017/deb test03012017/rpm test03012017/darwin test03012017/win test03012017/bin /home/user/go_workspace/src/github.com/user/test03012017/build/upload/build_id.txt make[1]: Entering directory '/home/user/go_workspace/src/github.com/user/test03012017/vendor/github.com/elastic/beats/dev-tools/packer' docker run --rm -v /home/user/go_workspace/src/github.com/user/test03012017/build:/build tudorg/go-daemon Fetching go-daemon Cloning into 'go-daemon'... Compiling for linux/amd64.. /tmp/cc9Yke9f.o: In function main':
god.c:(.text+0x2ca): warning: Using 'getgrnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
god.c:(.text+0x262): warning: Using 'getpwnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
Compiling for linux/i386..
/tmp/cc08rmPq.o: In function `main':
god.c:(.text+0x2ee): warning: Using 'getgrnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
god.c:(.text+0x287): warning: Using 'getpwnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
Copying to host..
docker build --rm=true -t tudorg/fpm /home/user/go_workspace/src/github.com/user/test03012017/vendor/github.com/elastic/beats/dev-tools/packer/docker/fpm-image
Sending build context to Docker daemon 2.048 kB
Step 1 : FROM ubuntu:16.04
---> e4415b714b62
Step 2 : MAINTAINER Tudor Golubenco tudor@elastic.co
---> Using cache
---> 5563b321f868
Step 3 : RUN apt-get update && apt-get install -y --no-install-recommends build-essential ruby-dev rpm zip dos2unix libgmp3-dev
---> Using cache
---> 0f14779ca09c
Step 4 : RUN gem install fpm
---> Using cache
---> 5def1a2c73a9
Successfully built 5def1a2c73a9
echo Creating DEB packages for test03012017
Creating DEB packages for test03012017
ARCH=386 BEAT=test03012017 BUILD_DIR=/home/user/go_workspace/src/github.com/user/test03012017/build BEAT_DIR=/home/user/go_workspace/src/github.com/user/test03012017 BUILDID= SNAPSHOT=yes /home/user/go_workspace/src/github.com/user/test03012017/vendor/github.com/elastic/beats/dev-tools/packer/platforms/debian/build.sh
/home/user/go_workspace/src/github.com/user/test03012017/vendor/github.com/elastic/beats/dev-tools/packer/platforms/debian/build.sh: 12: /home/user/go_workspace/src/github.com/user/test03012017/vendor/github.com/elastic/beats/dev-tools/packer/platforms/debian/build.sh: gotpl: not found
Makefile:13: recipe for target 'test03012017/deb' failed
make[1]: *** [test03012017/deb] Error 127
make[1]: Leaving directory '/home/user/go_workspace/src/github.com/user/test03012017/vendor/github.com/elastic/beats/dev-tools/packer'
vendor/github.com/elastic/beats/libbeat/scripts/Makefile:400: recipe for target 'package' failed
make: *** [package] Error 2

After the steps:

  export GOPATH=/home/user/go_workspace
  export PATH=$PATH:/usr/local/go/bin
  rm -rf /home/user/go_workspace/*
  cd /home/user/go_workspace/
  go get github.com/elastic/beats
  mkdir $GOPATH/src/github.com/user
  cd $GOPATH/src/github.com/user
  cookiecutter $GOPATH/src/github.com/elastic/beats/generate/beat
  cd /home/user/go_workspace/src/github.com/user/test03012017/
  make clean
  make setup
  make package

ruflin · January 3, 2017, 10:19am

What is the libbeat commit id you are on?

Aliaksandr_Zabrodski · January 12, 2017, 1:09pm

I rechecked the issue today using the following version:

# cd $GOPATH/src/github.com/elastic/beats/libbeat/
# git rev-parse --short HEAD
19e82d5

The error is:

make[1]: Entering directory '/home/user/go_workspace/src/github.com/user/test120117'
fatal: Not a git repository (or any of the parent directories): .git
# cross compile on ubuntu
docker run --rm \
        -v /home/user/go_workspace/src/github.com/user/test120117/vendor/github.com/elastic/beats/dev-tools/packer/xgo-scripts:/scripts \
        -v /home/user/go_workspace/src/github.com/user/test120117/vendor/github.com/elastic/beats/dev-tools/packer:/templates \
        -v /home/user/go_workspace/src/github.com/user:/source \
        -v /home/user/go_workspace/src/github.com/user/test120117/build:/build \
        -e PUREGO="yes" \
        -e PACK=test120117 \
        -e BEFORE_BUILD=before_build.sh \
        -e SOURCE=/source \
        -e TARGETS="linux/amd64 linux/386 windows/amd64 windows/386 darwin/amd64"  \
        -e BUILDID=  \
        tudorg/beats-builder  \
        github.com/user/test120117
Working directory=/go/src/github.com/user
Copying main source folder /source to folder /go/src/github.com/user
Fetching dependencies...
Execute /scripts/before_build.sh github.com/user/test120117
install -d -m 755 /build/homedir/scripts/
install -m 755 ./vendor/github.com/elastic/beats/libbeat/scripts/migrate_beat_config_1_x_to_5_0.py /build/homedir/scripts/
Compiling import_dashboards for linux/amd64
Compiling import_dashboards for linux/386
Compiling import_dashboards for windows/amd64
Compiling import_dashboards for windows/386
Compiling import_dashboards for darwin/amd64
Cloning into '/go/src/github.com/tsg/gotpl'...
make: Nothing to be done for `before-build'.
Compiling test120117 for linux/amd64...
Compiling test120117 for linux/386...
Compiling test120117 for windows-4.0/amd64...
Compiling test120117 for windows-4.0/386...
Compiling test120117 for darwin/amd64...
Build process completed
make[1]: Leaving directory '/home/user/go_workspace/src/github.com/user/test120117'
SNAPSHOT=yes  BUILDID=  BEAT_DIR=github.com/user/test120117 BUILD_DIR=/home/user/go_workspace/src/github.com/user/test120117/build make -C ./vendor/github.com/elastic/beats/dev-tools/packer test120117/deb test120117/rpm test120117/darwin test120117/win test120117/bin  /home/user/go_workspace/src/github.com/user/test120117/build/upload/build_id.txt
make[1]: Entering directory '/home/user/go_workspace/src/github.com/user/test120117/vendor/github.com/elastic/beats/dev-tools/packer'
docker run --rm -v /home/user/go_workspace/src/github.com/user/test120117/build:/build tudorg/go-daemon
Fetching go-daemon
Cloning into 'go-daemon'...
Compiling for linux/amd64..
/tmp/ccj0a8Vu.o: In function `main':
god.c:(.text+0x2ca): warning: Using 'getgrnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
god.c:(.text+0x262): warning: Using 'getpwnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
Compiling for linux/i386..
/tmp/cc9NmdkE.o: In function `main':
god.c:(.text+0x2ee): warning: Using 'getgrnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
god.c:(.text+0x287): warning: Using 'getpwnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
Copying to host..
docker build --rm=true -t tudorg/fpm /home/user/go_workspace/src/github.com/user/test120117/vendor/github.com/elastic/beats/dev-tools/packer/docker/fpm-image
Sending build context to Docker daemon 2.048 kB
Step 1 : FROM ubuntu:16.04
 ---> e4415b714b62
Step 2 : MAINTAINER Tudor Golubenco <tudor@elastic.co>
 ---> Using cache
 ---> 5563b321f868
Step 3 : RUN apt-get update &&     apt-get install -y --no-install-recommends         build-essential ruby-dev rpm zip dos2unix libgmp3-dev
 ---> Using cache
 ---> 0f14779ca09c
Step 4 : RUN gem install fpm
 ---> Using cache
 ---> 5def1a2c73a9
Successfully built 5def1a2c73a9
echo Creating DEB packages for test120117
Creating DEB packages for test120117
ARCH=386 BEAT=test120117 BUILD_DIR=/home/user/go_workspace/src/github.com/user/test120117/build BEAT_DIR=/home/user/go_workspace/src/github.com/user/test120117 BUILDID= SNAPSHOT=yes /home/user/go_workspace/src/github.com/user/test120117/vendor/github.com/elastic/beats/dev-tools/packer/platforms/debian/build.sh
/home/user/go_workspace/src/github.com/user/test120117/vendor/github.com/elastic/beats/dev-tools/packer/platforms/debian/build.sh: 12: /home/user/go_workspace/src/github.com/user/test120117/vendor/github.com/elastic/beats/dev-tools/packer/platforms/debian/build.sh: gotpl: not found
Makefile:14: recipe for target 'test120117/deb' failed
make[1]: *** [test120117/deb] Error 127
make[1]: Leaving directory '/home/user/go_workspace/src/github.com/user/test120117/vendor/github.com/elastic/beats/dev-tools/packer'
vendor/github.com/elastic/beats/libbeat/scripts/Makefile:397: recipe for target 'package' failed
make: *** [package] Error 2

ruflin · January 12, 2017, 1:51pm

That is a commit from yesterday

I think the important part is:

Some more ideas:

Can you run locally go get -u github.com/tsg/gotpl and run it again?
If that doesn't work, can you remove ALL your docker images and let them rebuild? I kind of have the feeling something is out of date

Can you share the content of your Makefile?

Aliaksandr_Zabrodski · January 13, 2017, 12:28pm

I cleared all docker images and containers, executed the go get -u github.com/tsg/gotpl and after the make package i got the same error:

Successfully built 929696f72b50
echo Creating DEB packages for test1301
Creating DEB packages for test1301
ARCH=386 BEAT=test1301 BUILD_DIR=/home/user/go_workspace/src/github.com/user/test1301/build BEAT_DIR=/home/user/go_workspace/src/github.com/user/test1301 BUILDID= SNAPSHOT=yes /home/user/go_workspace/src/github.com/user/test1301/vendor/github.com/elastic/beats/dev-tools/packer/platforms/debian/build.sh
/home/user/go_workspace/src/github.com/user/test1301/vendor/github.com/elastic/beats/dev-tools/packer/platforms/debian/build.sh: 12: /home/user/go_workspace/src/github.com/user/test1301/vendor/github.com/elastic/beats/dev-tools/packer/platforms/debian/build.sh: gotpl: not found
Makefile:14: recipe for target 'test1301/deb' failed
make[1]: *** [test1301/deb] Error 127
make[1]: Leaving directory '/home/user/go_workspace/src/github.com/user/test1301/vendor/github.com/elastic/beats/dev-tools/packer'
vendor/github.com/elastic/beats/libbeat/scripts/Makefile:397: recipe for target 'package' failed
make: *** [package] Error 2

Makefile:

BEATNAME=test1301
BEAT_DIR=github.com/user/test1301
BEAT_GOPATH=$(firstword $(subst :, ,${GOPATH}))
SYSTEM_TESTS=false
TEST_ENVIRONMENT=false
ES_BEATS?=./vendor/github.com/elastic/beats
GOPACKAGES=$(shell glide novendor)
PREFIX?=.

# Path to the libbeat Makefile
-include $(ES_BEATS)/libbeat/scripts/Makefile

# Initial beat setup
.PHONY: setup
setup: copy-vendor
        make update

# Copy beats into vendor directory
.PHONY: copy-vendor
copy-vendor:
        mkdir -p vendor/github.com/elastic/
        cp -R ${BEAT_GOPATH}/src/github.com/elastic/beats vendor/github.com/elastic/
        rm -rf vendor/github.com/elastic/beats/.git

.PHONY: git-init
git-init:
        git init
        git add README.md CONTRIBUTING.md
        git commit -m "Initial commit"
        git add LICENSE
        git commit -m "Add the LICENSE"
        git add .gitignore
        git commit -m "Add git settings"
        git add .
        git reset -- .travis.yml
        git commit -m "Add test1301"
        git add .travis.yml
        git commit -m "Add Travis CI"

# This is called by the beats packer before building starts
.PHONY: before-build
before-build:

# Collects all dependencies and then calls update
.PHONY: collect
collect:

ruflin · January 16, 2017, 2:17pm

There is a similar issue here: Building custom filebeat debian Still trying to figure out why this happens

For the Makefile: We just merged a PR and you need to change BEATNAME to BEAT_NAME and BEAT_DIR to BEAT_PATH. But this should not be related to this issue.

ruflin · January 17, 2017, 11:46am

I have here a PR in the working that changes the gotpl handling. This could solve your issue: https://github.com/elastic/beats/pull/3386

surajs · January 24, 2017, 1:01am

Sorry for the delayed reply. Thanks for the fix.

system · February 21, 2017, 1:01am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Packaging community beats Beats	7	1844	July 5, 2017
Beats make package fails with `gotpl: not found` error Beats	2	981	July 12, 2017
Make package fails for beats plugin Beats beats-development	2	582	July 27, 2018
"make package" broken for community beats? Beats	4	1193	January 25, 2017
Make filebeat error: cannot find package "github.com/elastic/beats/filebeat/beater" in any of: Beats filebeat	2	1743	September 6, 2017

Make package: yaml.v2': No such file or directory

Generates the package.yml file with all information needed to create packages

cross compile on ubuntu

Related topics