update the metricbeat.yml and run the metricbeat binary
This works locally, but in a production environment what is the best practice for deploying these? Right now, we're just using system packages to handle metricbeat but is there a simple way to add a new beat to the precompiled binaries? We could compile metricbeat (using the steps above) each time we redeploy, but that would be a lot of time wasted compiling.
I guess I could create my own binary and use that, but that would mean actively recompiling the binary when new versions of metricbeat are released.
Is there any solid way to handle this, or is using my own binaries the best option?
I think the best solution is to use the mage package functionality to make my own package, as described in the docs for creating a custom beat (here). I'm not sure if there is an easier way but this should be sufficient.
Your case is really interesting! Btw, what approach did you actually decide to try? Creating a custom Beat or extending Metricbeat with a new metricset? Also please have in mind that you can extend Metricbeat with a new Module. Module can contain multiple metricsets. Let me put some comments on the 2 different approaches:
Custom Beat: In this case you use Beat's code as an extrenal library and you build your Beat out of Beats' codebase. This is has the advantage of being simple and quick and you just maintain your own Beat. From updates' perspective, if sth new has come on Beat's side and you want to leverage it
on your custom Beat, then most probably you will need to update Beat's library. However I'm not sure if everything new on Beat's side can be used from a custom Beat.
Metricbeat Module: With this way you just fork Beats project and you extend it with your new module in a native way. This means that whenever you want new updates from the upstream project you just rebase your fork and you have all the new code on your forked project.
Both cases work, with the second being more smooth since it's actually how beats developers develop on daily basis (more or less ).
Regarding packaging, yes make package should be the way to since it will create all the nessecary artifacts for you.
Thanks for taking interest, I figured it was a bit of an odd scenario. I made a custom metricset (the terminology is a bit confusing sometimes haha) because statsd kinda worked but kinda didn't for what I wanted.
I created a build script that does the following:
git clone https://github.com/elastic/beats; cd beats;
This got me all set up for the project with my own metricset. Then I copy the files for the custom metricset (from my own repo) and put them in the beats/metricbeat/module/metricsd/server folder. After that, I just do the mage update; mage build; mage package and get my own binary. Then I upload the binary to our own S3 bucket.
The real tricky part for me has been how to safely deploy the binary (on AWS Linux). My strategy has been to do this:
yum -y install metricbeat
yum -y install metricbeat-oss-7.10.2-x86_64.rpm
This *kind of works, because it generally keeps all the systemd configs but with my own custom metricbeat. It has been causing problems recently and I haven't totally solved it, but this process works well enough (especially because I rarely have to update the custom metricset code). I do have no idea whats gonna happen when the yum metricbeat package changes verisons.
For what it's worth, I really didn't wanna manage a fork of the beats repo because its so much code and I only need to change a little bit (and i'd have to write my own systemd service).
EDIT
As expected, it broke when the system metricbeat package updated to v7.11.2. I just am completely ignoring the system package now and only installing the rpm and it's working great.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
).