Beats starting kit

Elastic Stack Beats
1

Hey beats team,

I'm really happy to hear the concept of Sysbeat at Is there interest in something like a MonitBeat?. Though, before that we may need to have more specific beats being implemented.

When I'm starting a new beat implementation (hello uwsgibeat https://github.com/mrkschan/uwsgibeat :slight_smile: ) I just find myself copy-n-pasting stuff here and there. Shall we have a starter kit to generate a project from a template like the way in https://github.com/audreyr/cookiecutter? Would https://github.com/elastic/beats-packer be a good place to house that?

2

I'd prefer to reduce some boilerplate required to start a beat. The less generated code (template), the less to maintain if things change.

But in meantime some template project to copy from is nice to have. Currently topbeat is a good starting point.

3

@mrkschan I think as soon as version 1.0 is out this is definitively something we should discuss. As @urso mentioned currently topbeat is a good project to clone and then start with your own implementation.

In the end all beats should have the same Makefile, config structure, .gitignore file etc or at least share the same commands. As soon as we have it unified in our existing beats, creating the template / repo should be easy.

@mrkschan As you created already more then one beat it is definitively good to hear from your side what would help you most to speed up and simplify the creation of new beats.

4

@mrkschan Side note: We have a list of community beats here: https://github.com/elastic/libbeat/blob/1.0.0-beta4/docs/communitybeats.asciidoc It is currently only in the beta4 branch but will be in the master branch soon. Feel free to open a pull request so your beats are added.

5

So far I can tell... the following can be generated easily by template.

  • .gitignore
  • main.go
  • Xbeat.go
  • config.go
  • Makefile
  • .travis.yml
  • scripts/generate-*.py
  • etc/Xbeat.yml
  • LICENSE (Apache 2?)

Perhaps, shall we just use cookiecutter to do so at first?

6

hm... number of files already.

At least scripts/generate-*.py I'd like to have somewhere in libbeat and reusable

etc/fields.yml missing. Use to generate docs/fields.asciidoc + used in integration tests to validate published events

Other files? README.md, CHANGELOG.md, docs/index.asciidoc

7

README - generated by Github init repo :smile:
CHANGELOG - Good point :slight_smile:
docs/index.asciidoc - Yes!
etc/fields.yml - Not sure since not many things can be shared, though a boilerplate is good.

8

@mrkschan I also have internal notes on this. I will go through it, clean it up and make it available as a libbeat issue, so we can directly discuss it there? What I intend to do is not only create a template but also make it possible to easily fetch updates from each beat (new default config, libbeat, ...)

9

For follow up discussion, see https://github.com/elastic/libbeat/issues/280