I've been setting up an ELK stack to collect various metrics. Most of them will come from hosts where I can't afford installing Logstash, so I plan to use Beats.
For logs, it's quite trivial, thanks to Filebeat, but I also want to ship data that are currently stored in RRD files (created by Munin, SmokePing…).
For those who don't know, RRD is a simplified database, with constant very efficient disk usage. A lot of data can fit in a small file and the granularity increases over time. The most recent metrics are quite precise (for example 5 minutes of granularity) and the most ancient metrics are reaggregated in 1 day buckets. There is a rock solid library to manipulate RRD files : rrdtools (http://oss.oetiker.ch/rrdtool/index.en.html).
I'd like to start developing a beat for shipping RRD data to ES or Logstash, using the libbeat framework. I'm an experienced developer (mostly Ruby) but I'm brand new to Go and its ecosystem.
I've noticed the beat-generator project, but there is a lot coming with it : Docker, Python virtualenv, Glide, Go, libbeat… and it's a bit overwhelming. Is it the best way to begin with Beats ? Or is it too young and not enough stable/documented for a Go beginner ?
My repository is here : https://github.com/jlecour/rrdbeat/ It is a bare result of the generator, and I can't "compile".
jlecour@x1:~/code/rrdbeat$ make go build main.go:8:2: cannot find package "rrdbeat/rrdbeat/beater" in any of: /usr/local/go/src/rrdbeat/rrdbeat/beater (from $GOROOT) /home/jlecour/code/src/rrdbeat/rrdbeat/beater (from $GOPATH) vendor/github.com/elastic/beats/libbeat/scripts/Makefile:63: recipe for target 'build' failed make: *** [build] Error 1 jlecour@x1:~/code/rrdbeat$ glide update [INFO] Downloading dependencies. Please wait... [INFO] Fetching updates for github.com/elastic/beats. [INFO] Setting version for github.com/elastic/beats to 40146f0616ff4bce05acbc0435c9828aa435bfd3. [INFO] Resolving imports [INFO] Fetching updates for github.com/satori/go.uuid. [INFO] Fetching rrdbeat/rrdbeat/beater into /home/jlecour/code/rrdbeat/vendor [ERROR] Error looking for rrdbeat/rrdbeat/beater: Cannot detect VCS [INFO] Fetching updates for github.com/nranchev/go-libGeoIP. [INFO] Fetching updates for github.com/urso/ucfg. [WARN] Download failed.
I'd appreciate all the guidance and advice you'd give , especially from @ruflin who seems to be the go-to person regarding this.