Combining multiple beats into a single beat

Elastic Stack Beats
1

Hi there,

I know this topic has been discussed before (and I'd add to it if it wasn't closed Combining multiple beats into a single agent). However, I'd like to investigate how hard it would be for me as a developer (with no go experience) to combine multiple beats into a single agent.

The main reason for this is that we plan to ask our customers to install beats in their environment. It would be FAR simpler for them if there's a single beat to install and configure, and depending on the use-case, enable or disable certain collectors. I'm a little confused why beats don't already work like this. Logstash, Kibana and Elasticsearch all use plugins to extend functionality, why not beats? All the outputs for the beats seem to be the same anyway (kafka, logstash, elasticsearch). Surely a plugin based approach becomes more sensible as more and more beats are published? What if someone wanted 10 different beat collectors on a single box?

Could I pretty please ask someone who knows the code to give a very rough outline how I could combine a few select beats into a single über-beat? Also, an indication of how hard it would be would be fantastic?

Many thanks!
Nick G

1 Like
2

Hi Nick,

Would be sufficient for you to use a tool like supervisord to manage multiple Beats? We are thinking to build a similar tool that would be able to fetch the Beats (offline or online) and run them, one in each process, depending on what the user configures to monitor (nginx logs, nginx metrics, etc).

Would be a concern for your customers to run multiple processes? Would you prefer more to combine the Beats together into a single executable?

What Beats would you be interested in combining together?

1 Like
3

Thanks for your reply Monica,

We ask our customers to install and configure beats on their systems. We're hoping to make the whole process as simple as possible.

I'm not so much concerned about whether the beats run as separate processes (or threads), but the administrative burden of configuring and running multiple beats.

Ideally, what I'm looking for is a way to achieve all of the following:

  • A single configuration file
  • A single log file for all the beats
  • A single package to install
  • A plugin architecture which, for people who run Logstash, Kibana or Elasticsearch are already familiar with, e.g install the main package, then pull down one or more plugins. I'd also be very keen to learn of a way to "bake in" multiple beats into the package, so our customers didn't even need to install plugins.
  • No requirement on the client installing and configuring a third party package (supervisord).

Part of the attraction of beats is their simplicity. If the tool you're thinking of building (similar to supervisord) provides some or all of the features above, I'd be very interested in trying it out!

At a minimum, we'd be looking at combining the following into a single beat:

  • filebeat
  • metricbeat
  • packetbeat
  • auditbeat
  • protologbeat

I understand I'm in no position to be demanding major architectural changes to such a critical component of your stack. However, it would be fantastic if something was done to make the process of managing multiple beats a bit (or a lot) easier.

Thanks again,
Nick G

2 Likes
4

This topic was automatically closed after 21 days. New replies are no longer allowed.