Make package: yaml.v2': No such file or directory

Yep the

make

command does not return any errors. Docker version is:

user@dev1:~$ docker --version
Docker version 1.12.3, build 6b644ec

I don't have the docker-compose installed. Should it be?

That is quite strange. I assume you have yaml.v2 only in the vendor directory of beats and not in the global GOPATH? Docker version also looks good. You are right that compose is not needed for the packaging part.

My tests are done on Mac OS Sierra. I'm getting the impression there is something going wrong with mounting the volumes into the docker container. As far as I can see you are on Debian?

I have some unknown yaml.v2.a file, not directory, under the gopath:

user@dev1:~$ $echo $GOPATH
    -bash: /home/user/go_workspace: Is a directory
user@dev1:~$ ls -ltr /home/user/go_workspace/pkg/linux_amd64/gopkg.in/yaml.v2.a
    -rw-r--r-- 1 user user 513782 Nov 22 15:29 /home/user/go_workspace/pkg/linux_amd64/gopkg.in/yaml.v2.a

Yep, trying to make it work on debian:

~$ user@dev1:~$ lsb_release -da
    No LSB modules are available.
    Distributor ID: Debian
    Description:    Debian GNU/Linux 8.3 (jessie)
    Release:        8.3
    Codename:       jessie

Sorry for the late reply. Is what you shared above the full output after make package or are there more lines? I would expect quite a few additional lines before.

I think I found the problem and here is the potential fix for it: https://github.com/elastic/beats/pull/3094

I was looking in the completely wrong place initially as I thought it happens directly after make package, but there are quite a lot of log lines in between. Not sure yet why it worked actually in the past :slight_smile:

Please let me know if you need me to reproduce the issue to get the make package output.

@Aliaksandr_Zabrodski It would be nice if you could keep an eye on the PR and retry as soon as it is merged and let me know if it fixes the issue for you.

Hi @ruflin,

I tried the PR and the problem is gone, but soon afterwards I run into another problem that the beat template file is not found, in my particular case I get the following message:

cp: cannot stat 'httpbeat.template.json': No such file or directory.

Please ignore the previous comment, the files where not in the correct place.
So I think the PR works as expected

Hi, @ruflin

I have tried the fix today and got another similar error at make package stage:

echo Creating DEB packages for test27122016
Creating DEB packages for test27122016
ARCH=386 BEAT=test27122016 BUILD_DIR=/home/user/go_workspace/src/github.com/user/test27122016/build BEAT_DIR=/home/user/go_workspace/src/github.com/user/test27122016 BUILDID= SNAPSHOT=yes /home/user/go_workspace/src/github.com/user/test27122016/vendor/github.com/elastic/beats/dev-tools/packer/platforms/debian/build.sh
/home/user/go_workspace/src/github.com/user/test27122016/vendor/github.com/elastic/beats/dev-tools/packer/platforms/debian/build.sh: 12: /home/user/go_workspace/src/github.com/user/test27122016/vendor/github.com/elastic/beats/dev-tools/packer/platforms/debian/build.sh: gotpl: not found
Makefile:13: recipe for target 'test27122016/deb' failed
make[1]: *** [test27122016/deb] Error 127
make[1]: Leaving directory '/home/user/go_workspace/src/github.com/user/test27122016/vendor/github.com/elastic/beats/dev-tools/packer'
vendor/github.com/elastic/beats/libbeat/scripts/Makefile:400: recipe for target 'package' failed
make: *** [package] Error 2

Extended make package output is below:

...    
Fetching dependencies...
Execute /scripts/before_build.sh github.com/user/test27122016
install -d -m 755 /build/homedir/scripts/
install -m 755 ./vendor/github.com/elastic/beats/libbeat/scripts/migrate_beat_config_1_x_to_5_0.py /build/homedir/scripts/
Compiling import_dashboards for linux/amd64
Compiling import_dashboards for linux/386
Compiling import_dashboards for windows/amd64
Compiling import_dashboards for windows/386
Compiling import_dashboards for darwin/amd64
Cloning into '/go/src/github.com/tsg/gotpl'...
make: Nothing to be done for `before-build'.
Compiling test27122016 for linux/amd64...
Compiling test27122016 for linux/386...
Compiling test27122016 for windows-4.0/amd64...
Compiling test27122016 for windows-4.0/386...
Compiling test27122016 for darwin/amd64...
Build process completed
make[1]: Leaving directory '/home/user/go_workspace/src/github.com/user/test27122016'
SNAPSHOT=yes BUILDID= BEAT_DIR=github.com/user/test27122016 BUILD_DIR=/home/user/go_workspace/src/github.com/user/test27122016/build make -C ./vendor/github.com/elastic/beats/dev-tools/packer test27122016/deb test27122016/rpm test27122016/darwin test27122016/win test27122016/bin /home/user/go_workspace/src/github.com/user/test27122016/build/upload/build_id.txt
make[1]: Entering directory '/home/user/go_workspace/src/github.com/user/test27122016/vendor/github.com/elastic/beats/dev-tools/packer'
docker run --rm -v /home/user/go_workspace/src/github.com/user/test27122016/build:/build tudorg/go-daemon
Fetching go-daemon
Cloning into 'go-daemon'...
Compiling for linux/amd64..
/tmp/ccHpxljV.o: In function `main':
god.c:(.text+0x2ca): warning: Using 'getgrnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
god.c:(.text+0x262): warning: Using 'getpwnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
Compiling for linux/i386..
/tmp/ccIl6uos.o: In function `main':
god.c:(.text+0x2ee): warning: Using 'getgrnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
god.c:(.text+0x287): warning: Using 'getpwnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
Copying to host..
docker build --rm=true -t tudorg/fpm /home/user/go_workspace/src/github.com/user/test27122016/vendor/github.com/elastic/beats/dev-tools/packer/docker/fpm-image
Sending build context to Docker daemon 2.048 kB
Step 1 : FROM ubuntu:16.04
 ---> e4415b714b62
Step 2 : MAINTAINER Tudor Golubenco <tudor@elastic.co>
 ---> Using cache
 ---> 5563b321f868
Step 3 : RUN apt-get update &&     apt-get install -y --no-install-recommends         build-essential ruby-dev rpm zip dos2unix libgmp3-dev
 ---> Using cache
 ---> 0f14779ca09c
Step 4 : RUN gem install fpm
 ---> Using cache
 ---> 5def1a2c73a9
Successfully built 5def1a2c73a9
echo Creating DEB packages for test27122016
Creating DEB packages for test27122016
ARCH=386 BEAT=test27122016 BUILD_DIR=/home/user/go_workspace/src/github.com/user/test27122016/build BEAT_DIR=/home/user/go_workspace/src/github.com/user/test27122016 BUILDID= SNAPSHOT=yes /home/user/go_workspace/src/github.com/user/test27122016/vendor/github.com/elastic/beats/dev-tools/packer/platforms/debian/build.sh
/home/user/go_workspace/src/github.com/user/test27122016/vendor/github.com/elastic/beats/dev-tools/packer/platforms/debian/build.sh: 12: /home/user/go_workspace/src/github.com/user/test27122016/vendor/github.com/elastic/beats/dev-tools/packer/platforms/debian/build.sh: gotpl: not found
Makefile:13: recipe for target 'test27122016/deb' failed
make[1]: *** [test27122016/deb] Error 127
make[1]: Leaving directory '/home/user/go_workspace/src/github.com/user/test27122016/vendor/github.com/elastic/beats/dev-tools/packer'
vendor/github.com/elastic/beats/libbeat/scripts/Makefile:400: recipe for target 'package' failed
make: *** [package] Error 2

We very recently change some code related to gotpl and these templates. Any chance you could again update to the most recent version of libbeat (master) and check again. Make sure to run make clean in advance.

Hm, i got the same error:

echo "Start building packages for test03012017"
Start building packages for test03012017
mkdir -p /home/user/go_workspace/src/github.com/user/test03012017/build/upload

Generates the package.yml file with all information needed to create packages

...

make[1]: Entering directory '/home/user/go_workspace/src/github.com/user/test03012017'
fatal: Not a git repository (or any of the parent directories): .git

cross compile on ubuntu

docker run --rm
-v /home/user/go_workspace/src/github.com/user/test03012017/vendor/github.com/elastic/beats/dev-tools/packer/xgo-scripts:/scripts
-v /home/user/go_workspace/src/github.com/user/test03012017/vendor/github.com/elastic/beats/dev-tools/packer:/templates
-v /home/user/go_workspace/src/github.com/user:/source
-v /home/user/go_workspace/src/github.com/user/test03012017/build:/build
-e PUREGO="yes"
-e PACK=test03012017
-e BEFORE_BUILD=before_build.sh
-e SOURCE=/source
-e TARGETS="linux/amd64 linux/386 windows/amd64 windows/386 darwin/amd64"
-e BUILDID=
tudorg/beats-builder
github.com/user/test03012017
Working directory=/go/src/github.com/user
Copying main source folder /source to folder /go/src/github.com/user
Fetching dependencies...
Execute /scripts/before_build.sh github.com/user/test03012017
install -d -m 755 /build/homedir/scripts/
install -m 755 ./vendor/github.com/elastic/beats/libbeat/scripts/migrate_beat_config_1_x_to_5_0.py /build/homedir/scripts/
Compiling import_dashboards for linux/amd64
Compiling import_dashboards for linux/386
Compiling import_dashboards for windows/amd64
Compiling import_dashboards for windows/386
Compiling import_dashboards for darwin/amd64
Cloning into '/go/src/github.com/tsg/gotpl'...
make: Nothing to be done for before-build'. Compiling test03012017 for linux/amd64... Compiling test03012017 for linux/386... Compiling test03012017 for windows-4.0/amd64... Compiling test03012017 for windows-4.0/386... Compiling test03012017 for darwin/amd64... Build process completed make[1]: Leaving directory '/home/user/go_workspace/src/github.com/user/test03012017' SNAPSHOT=yes BUILDID= BEAT_DIR=github.com/user/test03012017 BUILD_DIR=/home/user/go_workspace/src/github.com/user/test03012017/build make -C ./vendor/github.com/elastic/beats/dev-tools/packer test03012017/deb test03012017/rpm test03012017/darwin test03012017/win test03012017/bin /home/user/go_workspace/src/github.com/user/test03012017/build/upload/build_id.txt make[1]: Entering directory '/home/user/go_workspace/src/github.com/user/test03012017/vendor/github.com/elastic/beats/dev-tools/packer' docker run --rm -v /home/user/go_workspace/src/github.com/user/test03012017/build:/build tudorg/go-daemon Fetching go-daemon Cloning into 'go-daemon'... Compiling for linux/amd64.. /tmp/cc9Yke9f.o: In function main':
god.c:(.text+0x2ca): warning: Using 'getgrnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
god.c:(.text+0x262): warning: Using 'getpwnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
Compiling for linux/i386..
/tmp/cc08rmPq.o: In function `main':
god.c:(.text+0x2ee): warning: Using 'getgrnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
god.c:(.text+0x287): warning: Using 'getpwnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
Copying to host..
docker build --rm=true -t tudorg/fpm /home/user/go_workspace/src/github.com/user/test03012017/vendor/github.com/elastic/beats/dev-tools/packer/docker/fpm-image
Sending build context to Docker daemon 2.048 kB
Step 1 : FROM ubuntu:16.04
---> e4415b714b62
Step 2 : MAINTAINER Tudor Golubenco tudor@elastic.co
---> Using cache
---> 5563b321f868
Step 3 : RUN apt-get update && apt-get install -y --no-install-recommends build-essential ruby-dev rpm zip dos2unix libgmp3-dev
---> Using cache
---> 0f14779ca09c
Step 4 : RUN gem install fpm
---> Using cache
---> 5def1a2c73a9
Successfully built 5def1a2c73a9
echo Creating DEB packages for test03012017
Creating DEB packages for test03012017
ARCH=386 BEAT=test03012017 BUILD_DIR=/home/user/go_workspace/src/github.com/user/test03012017/build BEAT_DIR=/home/user/go_workspace/src/github.com/user/test03012017 BUILDID= SNAPSHOT=yes /home/user/go_workspace/src/github.com/user/test03012017/vendor/github.com/elastic/beats/dev-tools/packer/platforms/debian/build.sh
/home/user/go_workspace/src/github.com/user/test03012017/vendor/github.com/elastic/beats/dev-tools/packer/platforms/debian/build.sh: 12: /home/user/go_workspace/src/github.com/user/test03012017/vendor/github.com/elastic/beats/dev-tools/packer/platforms/debian/build.sh: gotpl: not found
Makefile:13: recipe for target 'test03012017/deb' failed
make[1]: *** [test03012017/deb] Error 127
make[1]: Leaving directory '/home/user/go_workspace/src/github.com/user/test03012017/vendor/github.com/elastic/beats/dev-tools/packer'
vendor/github.com/elastic/beats/libbeat/scripts/Makefile:400: recipe for target 'package' failed
make: *** [package] Error 2

After the steps:

  export GOPATH=/home/user/go_workspace
  export PATH=$PATH:/usr/local/go/bin
  rm -rf /home/user/go_workspace/*
  cd /home/user/go_workspace/
  go get github.com/elastic/beats
  mkdir $GOPATH/src/github.com/user
  cd $GOPATH/src/github.com/user
  cookiecutter $GOPATH/src/github.com/elastic/beats/generate/beat
  cd /home/user/go_workspace/src/github.com/user/test03012017/
  make clean
  make setup
  make package

:frowning: What is the libbeat commit id you are on?

I rechecked the issue today using the following version:

# cd $GOPATH/src/github.com/elastic/beats/libbeat/
# git rev-parse --short HEAD
19e82d5

The error is:

make[1]: Entering directory '/home/user/go_workspace/src/github.com/user/test120117'
fatal: Not a git repository (or any of the parent directories): .git
# cross compile on ubuntu
docker run --rm \
        -v /home/user/go_workspace/src/github.com/user/test120117/vendor/github.com/elastic/beats/dev-tools/packer/xgo-scripts:/scripts \
        -v /home/user/go_workspace/src/github.com/user/test120117/vendor/github.com/elastic/beats/dev-tools/packer:/templates \
        -v /home/user/go_workspace/src/github.com/user:/source \
        -v /home/user/go_workspace/src/github.com/user/test120117/build:/build \
        -e PUREGO="yes" \
        -e PACK=test120117 \
        -e BEFORE_BUILD=before_build.sh \
        -e SOURCE=/source \
        -e TARGETS="linux/amd64 linux/386 windows/amd64 windows/386 darwin/amd64"  \
        -e BUILDID=  \
        tudorg/beats-builder  \
        github.com/user/test120117
Working directory=/go/src/github.com/user
Copying main source folder /source to folder /go/src/github.com/user
Fetching dependencies...
Execute /scripts/before_build.sh github.com/user/test120117
install -d -m 755 /build/homedir/scripts/
install -m 755 ./vendor/github.com/elastic/beats/libbeat/scripts/migrate_beat_config_1_x_to_5_0.py /build/homedir/scripts/
Compiling import_dashboards for linux/amd64
Compiling import_dashboards for linux/386
Compiling import_dashboards for windows/amd64
Compiling import_dashboards for windows/386
Compiling import_dashboards for darwin/amd64
Cloning into '/go/src/github.com/tsg/gotpl'...
make: Nothing to be done for `before-build'.
Compiling test120117 for linux/amd64...
Compiling test120117 for linux/386...
Compiling test120117 for windows-4.0/amd64...
Compiling test120117 for windows-4.0/386...
Compiling test120117 for darwin/amd64...
Build process completed
make[1]: Leaving directory '/home/user/go_workspace/src/github.com/user/test120117'
SNAPSHOT=yes  BUILDID=  BEAT_DIR=github.com/user/test120117 BUILD_DIR=/home/user/go_workspace/src/github.com/user/test120117/build make -C ./vendor/github.com/elastic/beats/dev-tools/packer test120117/deb test120117/rpm test120117/darwin test120117/win test120117/bin  /home/user/go_workspace/src/github.com/user/test120117/build/upload/build_id.txt
make[1]: Entering directory '/home/user/go_workspace/src/github.com/user/test120117/vendor/github.com/elastic/beats/dev-tools/packer'
docker run --rm -v /home/user/go_workspace/src/github.com/user/test120117/build:/build tudorg/go-daemon
Fetching go-daemon
Cloning into 'go-daemon'...
Compiling for linux/amd64..
/tmp/ccj0a8Vu.o: In function `main':
god.c:(.text+0x2ca): warning: Using 'getgrnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
god.c:(.text+0x262): warning: Using 'getpwnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
Compiling for linux/i386..
/tmp/cc9NmdkE.o: In function `main':
god.c:(.text+0x2ee): warning: Using 'getgrnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
god.c:(.text+0x287): warning: Using 'getpwnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
Copying to host..
docker build --rm=true -t tudorg/fpm /home/user/go_workspace/src/github.com/user/test120117/vendor/github.com/elastic/beats/dev-tools/packer/docker/fpm-image
Sending build context to Docker daemon 2.048 kB
Step 1 : FROM ubuntu:16.04
 ---> e4415b714b62
Step 2 : MAINTAINER Tudor Golubenco <tudor@elastic.co>
 ---> Using cache
 ---> 5563b321f868
Step 3 : RUN apt-get update &&     apt-get install -y --no-install-recommends         build-essential ruby-dev rpm zip dos2unix libgmp3-dev
 ---> Using cache
 ---> 0f14779ca09c
Step 4 : RUN gem install fpm
 ---> Using cache
 ---> 5def1a2c73a9
Successfully built 5def1a2c73a9
echo Creating DEB packages for test120117
Creating DEB packages for test120117
ARCH=386 BEAT=test120117 BUILD_DIR=/home/user/go_workspace/src/github.com/user/test120117/build BEAT_DIR=/home/user/go_workspace/src/github.com/user/test120117 BUILDID= SNAPSHOT=yes /home/user/go_workspace/src/github.com/user/test120117/vendor/github.com/elastic/beats/dev-tools/packer/platforms/debian/build.sh
/home/user/go_workspace/src/github.com/user/test120117/vendor/github.com/elastic/beats/dev-tools/packer/platforms/debian/build.sh: 12: /home/user/go_workspace/src/github.com/user/test120117/vendor/github.com/elastic/beats/dev-tools/packer/platforms/debian/build.sh: gotpl: not found
Makefile:14: recipe for target 'test120117/deb' failed
make[1]: *** [test120117/deb] Error 127
make[1]: Leaving directory '/home/user/go_workspace/src/github.com/user/test120117/vendor/github.com/elastic/beats/dev-tools/packer'
vendor/github.com/elastic/beats/libbeat/scripts/Makefile:397: recipe for target 'package' failed
make: *** [package] Error 2

That is a commit from yesterday :slight_smile: :thumbsup:

I think the important part is:

Some more ideas:

  • Can you run locally go get -u github.com/tsg/gotpl and run it again?
  • If that doesn't work, can you remove ALL your docker images and let them rebuild? I kind of have the feeling something is out of date :frowning:

Can you share the content of your Makefile?

I cleared all docker images and containers, executed the go get -u github.com/tsg/gotpl and after the make package i got the same error:

Successfully built 929696f72b50
echo Creating DEB packages for test1301
Creating DEB packages for test1301
ARCH=386 BEAT=test1301 BUILD_DIR=/home/user/go_workspace/src/github.com/user/test1301/build BEAT_DIR=/home/user/go_workspace/src/github.com/user/test1301 BUILDID= SNAPSHOT=yes /home/user/go_workspace/src/github.com/user/test1301/vendor/github.com/elastic/beats/dev-tools/packer/platforms/debian/build.sh
/home/user/go_workspace/src/github.com/user/test1301/vendor/github.com/elastic/beats/dev-tools/packer/platforms/debian/build.sh: 12: /home/user/go_workspace/src/github.com/user/test1301/vendor/github.com/elastic/beats/dev-tools/packer/platforms/debian/build.sh: gotpl: not found
Makefile:14: recipe for target 'test1301/deb' failed
make[1]: *** [test1301/deb] Error 127
make[1]: Leaving directory '/home/user/go_workspace/src/github.com/user/test1301/vendor/github.com/elastic/beats/dev-tools/packer'
vendor/github.com/elastic/beats/libbeat/scripts/Makefile:397: recipe for target 'package' failed
make: *** [package] Error 2

Makefile:

BEATNAME=test1301
BEAT_DIR=github.com/user/test1301
BEAT_GOPATH=$(firstword $(subst :, ,${GOPATH}))
SYSTEM_TESTS=false
TEST_ENVIRONMENT=false
ES_BEATS?=./vendor/github.com/elastic/beats
GOPACKAGES=$(shell glide novendor)
PREFIX?=.

# Path to the libbeat Makefile
-include $(ES_BEATS)/libbeat/scripts/Makefile

# Initial beat setup
.PHONY: setup
setup: copy-vendor
        make update

# Copy beats into vendor directory
.PHONY: copy-vendor
copy-vendor:
        mkdir -p vendor/github.com/elastic/
        cp -R ${BEAT_GOPATH}/src/github.com/elastic/beats vendor/github.com/elastic/
        rm -rf vendor/github.com/elastic/beats/.git

.PHONY: git-init
git-init:
        git init
        git add README.md CONTRIBUTING.md
        git commit -m "Initial commit"
        git add LICENSE
        git commit -m "Add the LICENSE"
        git add .gitignore
        git commit -m "Add git settings"
        git add .
        git reset -- .travis.yml
        git commit -m "Add test1301"
        git add .travis.yml
        git commit -m "Add Travis CI"

# This is called by the beats packer before building starts
.PHONY: before-build
before-build:

# Collects all dependencies and then calls update
.PHONY: collect
collect:

There is a similar issue here: Building custom filebeat debian Still trying to figure out why this happens :frowning:

For the Makefile: We just merged a PR and you need to change BEATNAME to BEAT_NAME and BEAT_DIR to BEAT_PATH. But this should not be related to this issue.

I have here a PR in the working that changes the gotpl handling. This could solve your issue: https://github.com/elastic/beats/pull/3386

Sorry for the delayed reply. Thanks for the fix.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.