Beat or Metricbeat?

[shanim]

Hi,

A noob question here. I am trying to integrate metric generated by an application that we have built with ES. We currently use filebeat to parse the statistics files generated by the application but would like to consider developing either:

• a new beat that uses the metricbeat library
• a metricbeat module

Metrics can be dumped out by running a CLI command or by performing an API call. What is the recommended approach among the two options above?

[ruflin]

Let's assume you do metrics every 10s. If you write it to a file it's up to your service to write it in that frequency. If the app skips an filebeat will not tell you as it only harvests the event. Also if your app is under load and it goes up to 15s, filebeat will still just keep reading it. With accessing an endpoint from metricbeat on a predefined schedule, it's up to metricbeat to decide and configure the schedule. If the endpoint is not available, you get an error reported by metricbeat that something is not as expected. If you write to file, you can replay historical data.

Above I try to describe the difference and it depends on your use my. My general recommendation is if you are looking for only a snapshot of the data every x seconds, I would go with MB. If you more look for logs of events with different schedules FB could be a solution. But subscribing to events API is now also supported by MB modules, as the above is not either or but a continuum.

[shanim]

We were looking at either writing a beat (https://www.elastic.co/guide/en/beats/devguide/current/creating-beat-from-metricbeat.html) or a metricbeat module (https://www.elastic.co/guide/en/beats/devguide/current/creating-metricbeat-module.html) to replace filebeat in our design.

Is there a difference in the two approaches? I'm inclined to work on a Metricbeat module, but I'd like to understand the limitations if any

[ruflin]

The metricbeat module is less code and easier to maintain as you only need to maintain the module code and not have to create all the code for a beat and keep it up-to-date. But it gives you less flexibility. If your goal is to retrieve metrics from an endpoint, the module should be more then enough.