Kafkabeat - help

Elastic Stack Beats
1

I am trying to follow the blog guide about Kafka:


to start collecting some Kafka stats.

I have basic experience with Beats via using the yum repos for installation.
No Beats development experience.
When I run the make command as instructed in the guide, the following error:

glide update  --no-recursive
make: glide: Command not found
make: *** [init] Error 127

I guess there's an assumption that the reader has some knowledge that I personally don't have.
I haven't found any topics on the forums here regarding Kafkabeat, and there isn't much info on google either regarding that error.

Any help would be appreciated.

Thanks

2

This doesn't appear to be mentioned in the Kafkabeat documentation, but CONTRIBUTING.md in the Beats repository notes that you need to install glide.

3

@David_McClain Let me know if it worked with the installation of glide recommended by @magnusbaeck The non community beats have the dependencies checked in so no glide to just build it is needed. In the past we recommended for the community beats to use glide and that is when kafkabeat was built.

4

I'll be looking at it a little later this morning thanks.
Can you clarify on your phrase

so no glide to just build it is needed

Thanks

5

Non-community beats no longer depend on glide for building but community beats like Kafkabeat might still use glide.

6

Ok, I've installed Go, and Glide.
I'm on Centos 6.7.
I made sure that $GOPATH/github.com/gingerwizard exist (as per the README)
I also created $GOPATH/src , $GOPATH/bin , and $GOPATH/pkg

When I run

make init

again as per the README

I get this output resulting in an error:

make init
glide update  --no-recursive
[INFO]  Downloading dependencies. Please wait...
[INFO]  --> Fetching updates for github.com/elastic/beats.
[INFO]  --> Fetching updates for github.com/Shopify/sarama.
[INFO]  --> Fetching updates for github.com/davecgh/go-spew.
[INFO]  --> Fetching updates for github.com/eapache/queue.
[INFO]  --> Fetching updates for github.com/eapache/go-resiliency.
[INFO]  --> Fetching updates for github.com/golang/snappy.
[INFO]  --> Fetching updates for github.com/wvanbergen/kazoo-go.
[INFO]  --> Fetching updates for github.com/samuel/go-zookeeper.
[INFO]  --> Fetching updates for github.com/klauspost/crc32.
[INFO]  --> Setting version for github.com/elastic/beats to d8ca37efa8888ce624e2cfaa90107e79fd41be1e.
[INFO]  Exporting resolved dependencies...
[INFO]  --> Exporting github.com/elastic/beats
[INFO]  --> Exporting github.com/Shopify/sarama
[INFO]  --> Exporting github.com/golang/snappy
[INFO]  --> Exporting github.com/davecgh/go-spew
[INFO]  --> Exporting github.com/wvanbergen/kazoo-go
[INFO]  --> Exporting github.com/eapache/queue
[INFO]  --> Exporting github.com/samuel/go-zookeeper
[INFO]  --> Exporting github.com/eapache/go-resiliency
[INFO]  --> Exporting github.com/klauspost/crc32
[INFO]  Replacing existing vendor dependencies
[WARN]  Skipping lockfile generation because full dependency tree is not being calculated
make update
make[1]: Entering directory `/opt/beats/kafkabeat/kafkabeat'
bash ./vendor/github.com/elastic/beats/libbeat/scripts/update.sh kafkabeat github.com/gingerwizard/kafkabeat ./vendor/github.com/elastic/beats/libbeat
Beat name: kafkabeat
Beat path: ../kafkabeat
Start modifying beat
Update config file
Update fields
Traceback (most recent call last):
  File "./vendor/github.com/elastic/beats/libbeat/scripts/generate_fields_docs.py", line 129, in <module>
    fields_to_asciidoc(input, output, beat_name.title())
  File "./vendor/github.com/elastic/beats/libbeat/scripts/generate_fields_docs.py", line 99, in fields_to_asciidoc
    output.write("* <<exported-fields-{}>>\n".format(doc))
ValueError: zero length field name in format
make[1]: *** [update] Error 1
make[1]: Leaving directory `/opt/beats/kafkabeat/kafkabeat'
make: *** [init] Error 2

Thanks

7

Wild guess: You're using Python 2.6 and the script requires 2.7? Try replacing {} with {0}.

8

Wild and accurate. Thanks.

I would almost rather update my python.... That seems like a cleaner solution right?

9

Yes, Python 2.6 is quite old by now.

10

So, I have python 2.7 installed now alongside python 2.6.
Done in such as way that it doesn't mess up Centos (since 2.6 is a dependancy for Centos)

How would I go about telling the kafkabeat python files to use the right version?

env python -V is still the 2.6 version.
I have to go env python2.7 to get 2.7.

11

Update, I followed these steps:


To point yum to python2.6 and then created a symlink /usr/bin/python to point to my installation of python2.7.

So now the make init is working.
But make is not.

I'm troubleshooting various errors where a go script isn't finding the right code.

I also had to move the location of the gingerwizard code from:

$GOPATH/github.com/gingerwizard

to

$GOPATH/src/github.com/gingerwizard

Current errors I'm receiving are:

vendor/github.com/Shopify/sarama/message.go:10:2: cannot find package "github.com/eapache/go-xerial-snappy" in any of:
        /opt/beats/kafkabeat/go_work/src/github.com/gingerwizard/kafkabeat/vendor/github.com/eapache/go-xerial-snappy (vendor tree)
        /usr/lib/golang/src/github.com/eapache/go-xerial-snappy (from $GOROOT)
        /opt/beats/kafkabeat/go_work/src/github.com/eapache/go-xerial-snappy (from $GOPATH)
vendor/github.com/Shopify/sarama/broker.go:14:2: cannot find package "github.com/rcrowley/go-metrics" in any of:
        /opt/beats/kafkabeat/go_work/src/github.com/gingerwizard/kafkabeat/vendor/github.com/rcrowley/go-metrics (vendor tree)
        /usr/lib/golang/src/github.com/rcrowley/go-metrics (from $GOROOT)
        /opt/beats/kafkabeat/go_work/src/github.com/rcrowley/go-metrics (from $GOPATH)
make: *** [build] Error 1

I've found the rcrowley/go-metrics repo:

However, I can't find the eapache/go-xerial-snappy repo. It doesn't seem to exist anymore? Correction, I was only looking at eapache's pinned repos. https://github.com/eapache/go-xerial-snappy is what I was looking for.
12

Possible success!
After manually adding in the repos that it was complaining about, and moving the source code from where it originally told me to put it, I was able to get the make to work successfully and build the kafkabeat binary.

@magnusbaeck and @ruflin, do either of you suggest me posting this information as an issue onto the Kafkabeat repo? Or is that technically being phased out with the coming of Metricbeat in 5.0?

Thanks for all your help today.

13

IMHO Kafkabeat's documentation should indicate that glide is a requirement and Beats should list Python 2.7 (for the 2.x series ) or IIRC 3.1 (for the 3.x series) as a dependency and/or bail out with a reasonable error message.

14

Beats already lists the Python 2.7 dependency here: https://github.com/elastic/beats/blob/master/CONTRIBUTING.md#update-scripts Most scripts "should" also work on Python 3.x but we all run 2.7 so this is the one really tested.

About Kafkabeat and Metricbeat: We hope to add Kafka module to metricbeat in the near future but this is not the case yet. It is worth opening a PR against Kafkabeat to list this requirement. If you are interested in having Kafka module in Metricbeat, any help is welcome :slight_smile:

15

Thanks.
The python version was an oversight for me. However, it was a bit weird trying to compile it on Centos as Centos6.7 requires Python2.6 and you can't upgrade the system version.

The main issues I had where I thought the Kafkabeat documentation was even a little 'off' would be where it depends on some other repos that weren't included and I had to use the error messages to find then and clone them down.
Maybe I wasn't doing something right, but it was definitely a hurdle.

16

It is definitively not as easy as it should be. We spent lots of time in the official beats to make it as easy as possible and improved the beat-generator to also make it easier for community beats. Kafkabeat was created quite some time ago (in beats calculation of time) before this existed. For a community beat to catch up on all this is quite hard as it requires lots of efforts. That is also one reason we try to get these beats as modules into metricbeat so the community does not have to do the burden of the maintenance.

Let me know in case you hit similar issues with the official beats.

17

This topic was automatically closed after 21 days. New replies are no longer allowed.