Metricbeat using notification to collect data?

Right now, metricbeat uses polling to collect data from source, is there a way to use notification-based way? For example, it registers with events from source and in the callback of the event, extract data from event and push them to ES.

The problem with polling is that we may have huge traffic of data when polling. What is the usual practice to handle this huge traffic.

right now metricbeat is poll only. We're thinking about adding support for statsd and others.

The problem with polling is that we may have huge traffic of data when polling. What is the usual practice to handle this huge traffic.

Hm... weird. One argument/advantage of polling is, the collector has full control when sending data, while a push based collector (if it's not doing additional aggregations) does not. Potential problem with polling is all services/metrics being collect at same point in time -> short bursts of traffic/load generated every now and then.

We are collecting topology based data. They won't change very often. So polling might be overkill. Push + on demand polling might be a good choice to reduce traffic.

This sounds more like you are sending events/logs on topology updates. Maybe a job for filebeat? Filebeat can read json documents from log files. Some rudimentary UDP input has been added to filebeat recently.

Thanks, Steffen! Will explore!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.