# A beat for RRD files? (or How to start a beat, by a go-newbie?) **URL:** \ **Category:** Beats\ **Created:** [June 3, 2016, 10:11am UTC](https://discuss.elastic.co/t/a-beat-for-rrd-files-or-how-to-start-a-beat-by-a-go-newbie/51787 "2016-06-03T10:11:14Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Jeremy\_Lecour](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jeremy_lecour/32/1416_2.png) [@Jeremy\_Lecour](https://discuss.elastic.co/u/Jeremy_Lecour)\ **Post date:** [June 3, 2016, 10:11am UTC](https://discuss.elastic.co/t/a-beat-for-rrd-files-or-how-to-start-a-beat-by-a-go-newbie/51787/1 "2016-06-03T10:11:14Z")
Hi there, I've been setting up an ELK stack to collect various metrics. Most of them will come from hosts where I can't afford installing Logstash, so I plan to use Beats. For logs, it's quite trivial, thanks to Filebeat, but I also want to ship data that are currently stored in RRD files (created by Munin, SmokePing…). For those who don't know, RRD is a simplified database, with constant very efficient disk usage. A lot of data can fit in a small file and the granularity increases over time. The most recent metrics are quite precise (for example 5 minutes of granularity) and the most ancient metrics are reaggregated in 1 day buckets. There is a rock solid library to manipulate RRD files : **rrdtools** ([http://oss.oetiker.ch/rrdtool/index.en.html](http://oss.oetiker.ch/rrdtool/index.en.html)). I'd like to start developing a beat for shipping RRD data to ES or Logstash, using the **libbeat** framework. I'm an experienced developer (mostly Ruby) but I'm brand new to Go and its ecosystem. I've noticed the [beat-generator](https://github.com/elastic/beat-generator) project, but there is a lot coming with it : Docker, Python virtualenv, Glide, Go, libbeat… and it's a bit overwhelming. Is it the best way to begin with Beats ? Or is it too young and not enough stable/documented for a Go beginner ? My repository is here : [https://github.com/jlecour/rrdbeat/](https://github.com/jlecour/rrdbeat/) It is a bare result of the generator, and I can't "compile". ``` jlecour@x1:~/code/rrdbeat$ make go build main.go:8:2: cannot find package "rrdbeat/rrdbeat/beater" in any of: /usr/local/go/src/rrdbeat/rrdbeat/beater (from $GOROOT) /home/jlecour/code/src/rrdbeat/rrdbeat/beater (from $GOPATH) vendor/github.com/elastic/beats/libbeat/scripts/Makefile:63: recipe for target 'build' failed make: *** [build] Error 1 jlecour@x1:~/code/rrdbeat$ glide update [INFO] Downloading dependencies. Please wait... [INFO] Fetching updates for github.com/elastic/beats. [INFO] Setting version for github.com/elastic/beats to 40146f0616ff4bce05acbc0435c9828aa435bfd3. [INFO] Resolving imports [INFO] Fetching updates for github.com/satori/go.uuid. [INFO] Fetching rrdbeat/rrdbeat/beater into /home/jlecour/code/rrdbeat/vendor [ERROR] Error looking for rrdbeat/rrdbeat/beater: Cannot detect VCS [INFO] Fetching updates for github.com/nranchev/go-libGeoIP. [INFO] Fetching updates for github.com/urso/ucfg. [WARN] Download failed. ``` I'd appreciate all the guidance and advice you'd give 🙂, especially from @ruflin who seems to be the go-to person regarding this. Thanks. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [June 6, 2016, 10:52am UTC](https://discuss.elastic.co/t/a-beat-for-rrd-files-or-how-to-start-a-beat-by-a-go-newbie/51787/2 "2016-06-06T10:52:43Z")
Hi Jeremy Glad to hear you are building a beat. I think what went wrong here is setting the full go path to your repository. The line here ([https://github.com/jlecour/rrdbeat/blob/master/main.go#L8](https://github.com/jlecour/rrdbeat/blob/master/main.go#L8)) should be `github.com/jlecour/rrdbeat/beater` I'm not sure if here something went wrong during the generation (did you set your github name?) or if it si a problem in the generator. Normally it should work 🙂 This is definitively the right way to start with a beat. I'm currently working on making it even easier. Let me know if changes above work. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [June 6, 2016, 12:30pm UTC](https://discuss.elastic.co/t/a-beat-for-rrd-files-or-how-to-start-a-beat-by-a-go-newbie/51787/3 "2016-06-06T12:30:51Z")
@Jeremy_Lecour There are probably more incorrect paths, as the path was probably not set correctly during the setup. See also here: [https://github.com/jlecour/rrdbeat/blob/master/beater/rrdbeat.go#L12](https://github.com/jlecour/rrdbeat/blob/master/beater/rrdbeat.go#L12) Best is to search for it and replace it. Or in case you haven't made changes to the repo yet, you could generate a new version and make sure the path is correct. Let me know in case you hit some more issues so we can fix it. ---
**Author:** ![Jeremy\_Lecour](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jeremy_lecour/32/1416_2.png) [@Jeremy\_Lecour](https://discuss.elastic.co/u/Jeremy_Lecour)\ **Post date:** [June 7, 2016, 9:14pm UTC](https://discuss.elastic.co/t/a-beat-for-rrd-files-or-how-to-start-a-beat-by-a-go-newbie/51787/4 "2016-06-07T21:14:55Z")
Hi @ruflin and thanks for your time. I've recreated the whole thing from scratch. I get `github.com/jlecour/rrdbeat/beater`, as you can see here : [https://github.com/jlecour/rrdbeat/blob/master/main.go#L8](https://github.com/jlecour/rrdbeat/blob/master/main.go#L8) `make init` and `make commit` displayed no error, but `make` displays this : ``` ~/code/go/github.com/jlecour/rrdbeat → make go build main.go:8:2: cannot find package "github.com/jlecour/rrdbeat/beater" in any of: /usr/local/Cellar/go/1.6.2/libexec/src/github.com/jlecour/rrdbeat/beater (from $GOROOT) /Users/jlecour/code/go/src/github.com/jlecour/rrdbeat/beater (from $GOPATH) make: *** [build] Error 1 ``` Thanks again. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [June 8, 2016, 6:56am UTC](https://discuss.elastic.co/t/a-beat-for-rrd-files-or-how-to-start-a-beat-by-a-go-newbie/51787/5 "2016-06-08T06:56:27Z")
I just made a fork of your repo and tried it locally. All seems to work as expected. I think there is an issue with where you have placed your sources: You have your source code in: ```auto ~/code/go/github.com/jlecour/rrdbeat ``` But the GOPATH is: ```auto ~/code/go/src/github.com/jlecour/rrdbeat ``` You are missing the **src** in your path. Setting up the GOPATH and where to place the sources is often confusing when getting started with Golang ☹ Placing all your code under src should get it working. ---
**Author:** ![Jeremy\_Lecour](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jeremy_lecour/32/1416_2.png) [@Jeremy\_Lecour](https://discuss.elastic.co/u/Jeremy_Lecour)\ **Post date:** [June 9, 2016, 7:37am UTC](https://discuss.elastic.co/t/a-beat-for-rrd-files-or-how-to-start-a-beat-by-a-go-newbie/51787/6 "2016-06-09T07:37:34Z")
Thanks @ruflin I've managed to compile the boilerplate example. Now I have to find how to use the [external Go library to use RRD files](https://github.com/ziutek/rrd/). I'll keep the repository public and will report back when something usable is available. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [June 9, 2016, 11:12am UTC](https://discuss.elastic.co/t/a-beat-for-rrd-files-or-how-to-start-a-beat-by-a-go-newbie/51787/7 "2016-06-09T11:12:02Z")
Cool, ping me if you hit some further issues. ---
**Author:** ![Jeremy\_Lecour](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jeremy_lecour/32/1416_2.png) [@Jeremy\_Lecour](https://discuss.elastic.co/u/Jeremy_Lecour)\ **Post date:** [June 11, 2016, 10:07am UTC](https://discuss.elastic.co/t/a-beat-for-rrd-files-or-how-to-start-a-beat-by-a-go-newbie/51787/8 "2016-06-11T10:07:52Z")
I've managed (code not pushed yet) to use the RRD library. Now I have to find how to use a registry (like Filebeat does) to keep track of RRD files and offsets. Is there some documentation or extracted code about that or I have to dive deep in Filebeat's source code (which seems a bit complicated, with lots of parts) ? Thanks. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [June 13, 2016, 7:51am UTC](https://discuss.elastic.co/t/a-beat-for-rrd-files-or-how-to-start-a-beat-by-a-go-newbie/51787/9 "2016-06-13T07:51:08Z")
Unfortunately the registry is not enough abstracted yet that it could just be reused. So it is probably best to copy the code from here: [https://github.com/elastic/beats/blob/master/filebeat/crawler/registrar.go#L198](https://github.com/elastic/beats/blob/master/filebeat/crawler/registrar.go#L198) There are some special tricks inside to properly work on Windows. Some notes on the long term development: We introduced the `data.path` recently as we expect more beats to need to persist data / state. One of the ideas is to have a generic support for the registry in libbeat (obviously not the case yet). Second TBH it would be very nice if RRD would just be one more prospector in filebeat, like an extension. I'm in the process on abstracting more and more parts of the prospector and harvester to make it easier to extend filebeat (like you can do with module in Metricbeat), but that is also not right around the corner yet. ---
**Author:** ![Jeremy\_Lecour](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jeremy_lecour/32/1416_2.png) [@Jeremy\_Lecour](https://discuss.elastic.co/u/Jeremy_Lecour)\ **Post date:** [June 15, 2016, 1:01pm UTC](https://discuss.elastic.co/t/a-beat-for-rrd-files-or-how-to-start-a-beat-by-a-go-newbie/51787/10 "2016-06-15T13:01:56Z")
Using filebeat directly makes sense. I guess I'll cook a simplified version of this for the moment and kind of wait for a more robust solution. Thanks for the help. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 24, 2016, 10:11am UTC](https://discuss.elastic.co/t/a-beat-for-rrd-files-or-how-to-start-a-beat-by-a-go-newbie/51787/11 "2016-06-24T10:11:14Z")
This topic was automatically closed after 21 days. New replies are no longer allowed.