A beat for RRD files? (or How to start a beat, by a go-newbie?)

Elastic Stack Beats
1

Hi there,

I've been setting up an ELK stack to collect various metrics. Most of them will come from hosts where I can't afford installing Logstash, so I plan to use Beats.

For logs, it's quite trivial, thanks to Filebeat, but I also want to ship data that are currently stored in RRD files (created by Munin, SmokePing…).

For those who don't know, RRD is a simplified database, with constant very efficient disk usage. A lot of data can fit in a small file and the granularity increases over time. The most recent metrics are quite precise (for example 5 minutes of granularity) and the most ancient metrics are reaggregated in 1 day buckets. There is a rock solid library to manipulate RRD files : rrdtools (http://oss.oetiker.ch/rrdtool/index.en.html).

I'd like to start developing a beat for shipping RRD data to ES or Logstash, using the libbeat framework. I'm an experienced developer (mostly Ruby) but I'm brand new to Go and its ecosystem.

I've noticed the beat-generator project, but there is a lot coming with it : Docker, Python virtualenv, Glide, Go, libbeat… and it's a bit overwhelming. Is it the best way to begin with Beats ? Or is it too young and not enough stable/documented for a Go beginner ?

My repository is here : https://github.com/jlecour/rrdbeat/ It is a bare result of the generator, and I can't "compile".

jlecour@x1:~/code/rrdbeat$ make
go build
main.go:8:2: cannot find package "rrdbeat/rrdbeat/beater" in any of:
	/usr/local/go/src/rrdbeat/rrdbeat/beater (from $GOROOT)
	/home/jlecour/code/src/rrdbeat/rrdbeat/beater (from $GOPATH)
vendor/github.com/elastic/beats/libbeat/scripts/Makefile:63: recipe for target 'build' failed
make: *** [build] Error 1

jlecour@x1:~/code/rrdbeat$ glide update
[INFO] Downloading dependencies. Please wait...
[INFO] Fetching updates for github.com/elastic/beats.
[INFO] Setting version for github.com/elastic/beats to 40146f0616ff4bce05acbc0435c9828aa435bfd3.
[INFO] Resolving imports
[INFO] Fetching updates for github.com/satori/go.uuid.
[INFO] Fetching rrdbeat/rrdbeat/beater into /home/jlecour/code/rrdbeat/vendor
[ERROR] Error looking for rrdbeat/rrdbeat/beater: Cannot detect VCS
[INFO] Fetching updates for github.com/nranchev/go-libGeoIP.
[INFO] Fetching updates for github.com/urso/ucfg.
[WARN] Download failed.

I'd appreciate all the guidance and advice you'd give :slight_smile:, especially from @ruflin who seems to be the go-to person regarding this.

Thanks.

2

Hi Jeremy

Glad to hear you are building a beat. I think what went wrong here is setting the full go path to your repository. The line here (https://github.com/jlecour/rrdbeat/blob/master/main.go#L8) should be github.com/jlecour/rrdbeat/beater

I'm not sure if here something went wrong during the generation (did you set your github name?) or if it si a problem in the generator. Normally it should work :slight_smile:

This is definitively the right way to start with a beat. I'm currently working on making it even easier. Let me know if changes above work.

3

@Jeremy_Lecour There are probably more incorrect paths, as the path was probably not set correctly during the setup. See also here: https://github.com/jlecour/rrdbeat/blob/master/beater/rrdbeat.go#L12 Best is to search for it and replace it. Or in case you haven't made changes to the repo yet, you could generate a new version and make sure the path is correct.

Let me know in case you hit some more issues so we can fix it.

4

Hi @ruflin and thanks for your time.

I've recreated the whole thing from scratch.
I get github.com/jlecour/rrdbeat/beater, as you can see here : https://github.com/jlecour/rrdbeat/blob/master/main.go#L8

make init and make commit displayed no error, but make displays this :

~/code/go/github.com/jlecour/rrdbeat → make
go build
main.go:8:2: cannot find package "github.com/jlecour/rrdbeat/beater" in any of:
	/usr/local/Cellar/go/1.6.2/libexec/src/github.com/jlecour/rrdbeat/beater (from $GOROOT)
	/Users/jlecour/code/go/src/github.com/jlecour/rrdbeat/beater (from $GOPATH)
make: *** [build] Error 1

Thanks again.

5

I just made a fork of your repo and tried it locally. All seems to work as expected. I think there is an issue with where you have placed your sources:

You have your source code in:

~/code/go/github.com/jlecour/rrdbeat

But the GOPATH is:

~/code/go/src/github.com/jlecour/rrdbeat

You are missing the src in your path. Setting up the GOPATH and where to place the sources is often confusing when getting started with Golang :frowning: Placing all your code under src should get it working.

6

Thanks @ruflin I've managed to compile the boilerplate example.

Now I have to find how to use the external Go library to use RRD files.

I'll keep the repository public and will report back when something usable is available.

7

Cool, ping me if you hit some further issues.

8

I've managed (code not pushed yet) to use the RRD library.

Now I have to find how to use a registry (like Filebeat does) to keep track of RRD files and offsets.
Is there some documentation or extracted code about that or I have to dive deep in Filebeat's source code (which seems a bit complicated, with lots of parts) ?

Thanks.

9

Unfortunately the registry is not enough abstracted yet that it could just be reused. So it is probably best to copy the code from here: https://github.com/elastic/beats/blob/master/filebeat/crawler/registrar.go#L198 There are some special tricks inside to properly work on Windows.

Some notes on the long term development: We introduced the data.path recently as we expect more beats to need to persist data / state. One of the ideas is to have a generic support for the registry in libbeat (obviously not the case yet). Second TBH it would be very nice if RRD would just be one more prospector in filebeat, like an extension. I'm in the process on abstracting more and more parts of the prospector and harvester to make it easier to extend filebeat (like you can do with module in Metricbeat), but that is also not right around the corner yet.

10

Using filebeat directly makes sense. I guess I'll cook a simplified version of this for the moment and kind of wait for a more robust solution.

Thanks for the help.

11

This topic was automatically closed after 21 days. New replies are no longer allowed.