Lsbeat blog instructions fail

Elastic Stack Beats
1

Hello!

The blog instructions at https://www.elastic.co/blog/build-your-own-beat need to be updated.

First, the current directory structure of a skeleton Beat using cookiecutter is the following:

  .
    ├── CONTRIBUTING.md
    ├── LICENSE
    ├── Makefile
    ├── README.md
    ├── _meta
    │   ├── beat.yml
    │   └── fields.yml
    ├── beater
    │   └── mybeat.go
    ├── config
    │   ├── config.go
    │   └── config_test.go
    ├── docs
    │   └── index.asciidoc
    ├── main.go
    ├── main_test.go
    ├── mybeat.template.json
    └── tests
        └── system
            ├── config
            │   └── mybeat.yml.j2
            ├── mybeat.py
            ├── requirements.txt
            └── test_base.py

    7 directories, 17 files

Also, the instruction git checkout 5.1 causes an error when running make setup for the first time:

File "./vendor/github.com/elastic/beats/libbeat/scripts/generate_index_pattern.py", line 38
    print "ERROR: Field", path, "is duplicated. Please delete it and try again. Fields already are", unique_fields
                           ^
SyntaxError: Missing parentheses in call to 'print'

This error does not happen if the master branch is checked out.

John

2

That seems to be our constant issue related to Python 2.7 and 3.x. We do all our work and testing on 2.7. Unfortunately we state (well hidden) in our Contributor guide (https://github.com/elastic/beats/blob/master/CONTRIBUTING.md#update-scripts) >= 2.7.9 but we should state >= 2.7.9 and < 3.0 even though we try to keep the two compatible. Also in the blog post we don't have any Python requirements unfortunately.

As this problem will keep us bugging I see some potential solutions:

  • We update our requirement to python 3.0. Problem is that lots of distros still ship 2.7 :frowning:
  • We find a way to automatically check compatibilty and perhaps even automate compatibility. I found this one here: https://docs.python.org/2/library/2to3.html
  • Use virtualenv to also install a specific python version. We use virtualenv for most things, so this could help. Need to check if this is possible.

I opened this PR here to see if the automated way works: https://github.com/elastic/beats/pull/3408

3

Only indirectly related to this, but an other thing which could be helpful for us is introducing something like a check-environment script that checks if the correct versions of the tools like golang or python are installed, because that is an issue we hit frequently.