Create new beat based on libbeat 6.1?

[cyrilleverrier]

Hi,

What is/will be the preferred way of creating a new go application that uses libbeat 6.1?

1. The official documentation says to generate the new project using the cookie cutter template:
   https://www.elastic.co/guide/en/beats/devguide/6.1/setting-up-beat.html

make setup just copies the content of the libbeat from GOPATH to the vendor folder.

Drawbacks: vendor/github.com/elastic/ is not managed by govendor tool

2. use govendor tool
   I followed the same setup as https://github.com/elastic/apm-server

Drawbacks: needs to hack the vendor/vendor.json file to update the origin.
It doesn't work out of the box

My steps:

govendor fetch github.com/elastic/beats/...@$(BEATS_VERSION)
govendor fetch github.com/elastic/beats/libbeat/kibana/@$(BEATS_VERSION)
scripts/patch_vendorjson.py
BEATS_VERSION=$(BEATS_VERSION) scripts/update_beats.sh
scripts/patch_vendorjson.py --done
BEATS_VERSION=$(BEATS_VERSION) scripts/update_beats.sh

scripts/patch_vendorjson.py patches the vendor.json (created by govendor fetch github.com/elastic/beats/...) to set the origin of all packages to github.com/elastic/beats/vendor/github.com/xxx/xxx)

Without this patch, the script update_beats.sh does nothing

However, there are still some issues.
For instance, make set_version 1.2.3 does not work because it expects set_version python file to be in ./vendor/github.com/elastic/beats/dev-tools/set_version. But the file is now in ./_beats/dev-tools/set_version

Both solutions no longer work out of the box!
I can submit PR but I need guidance on the best way to create/update new projects.

[ruflin]

Let me start by sharing few thoughts on why I did the script for apm-server. For me it was important to separate the golang files from what I would call the framework part. The go dependencies should be managed by whatever dependency tool is used. For the framework part we need some special inclusion / exclusion to make sure all needed files are there but not too many.

For the apm-server case the set-version is not relevant as it is tied to the Elastic Stack version. But I remember in your case you need to overwrite it, so yes this part is broken and we should find a fix for it. Potentially a variable that defines where the script is could help (as we did for quite a few other things).

Moving forward I think we should make the beats-update script part of generator and find some tricks to also make the set-version work again. More then happy to review PR's on this I wanted to do the first part myself for some time but never got to it.