Content Field - Number of **NEW** Log Lines Harvested

Can we have filebeat share the no. of new log lines

  • It considered for processing
  • It matched/filtered

These numbers can then be used in multiple ways to assess the health of an application.

Here are very simple use cases:
Case-1) Application is running fine, but log lines are not emitted at all due to unknown issue
Case-2) Application got hung, so init.d status check won't catch as init.d sees the process is still running
Case-3) Application is running at reduced speed than normal - may be due to a service/server the app depends on has become slow

Solution-Step1) If no. of new log lines that harvester found (or considered) for every harvester run is shared out, we could then use that count to be sent to Graphite

Solution-Step2) Alert on this metric being zero for first two cases. Alert on this metric with a threshold for case-3

I have read Filebeat documentation available @ https://www.elastic.co/guide/en/beats/filebeat/current/index.html, also searched existing questions on this forums and thru general Google search. I see that log line offset is shared (per https://www.elastic.co/guide/en/beats/filebeat/current/exported-fields-log.html), but not no. of new log lines which is what we need.

Thanks,
Laxmi

Have you given the existing Beats Monitoring Metrics a chance?

I think it's a good fit to cover all the use cases you're interested in, out of the box.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.