Content Field - Number of **NEW** Log Lines Harvested

Elastic Stack Beats
1

Can we have filebeat share the no. of new log lines

  • It considered for processing
  • It matched/filtered

These numbers can then be used in multiple ways to assess the health of an application.

Here are very simple use cases:
Case-1) Application is running fine, but log lines are not emitted at all due to unknown issue
Case-2) Application got hung, so init.d status check won't catch as init.d sees the process is still running
Case-3) Application is running at reduced speed than normal - may be due to a service/server the app depends on has become slow

Solution-Step1) If no. of new log lines that harvester found (or considered) for every harvester run is shared out, we could then use that count to be sent to Graphite

Solution-Step2) Alert on this metric being zero for first two cases. Alert on this metric with a threshold for case-3

I have read Filebeat documentation available @ https://www.elastic.co/guide/en/beats/filebeat/current/index.html, also searched existing questions on this forums and thru general Google search. I see that log line offset is shared (per https://www.elastic.co/guide/en/beats/filebeat/current/exported-fields-log.html), but not no. of new log lines which is what we need.

Thanks,
Laxmi

2

Have you given the existing Beats Monitoring Metrics a chance?

I think it's a good fit to cover all the use cases you're interested in, out of the box.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.