Can we have filebeat share the no. of new log lines
- It considered for processing
- It matched/filtered
These numbers can then be used in multiple ways to assess the health of an application.
Here are very simple use cases:
Case-1) Application is running fine, but log lines are not emitted at all due to unknown issue
Case-2) Application got hung, so init.d status check won't catch as init.d sees the process is still running
Case-3) Application is running at reduced speed than normal - may be due to a service/server the app depends on has become slow
Solution-Step1) If no. of new log lines that harvester found (or considered) for every harvester run is shared out, we could then use that count to be sent to Graphite
Solution-Step2) Alert on this metric being zero for first two cases. Alert on this metric with a threshold for case-3
I have read Filebeat documentation available @ https://www.elastic.co/guide/en/beats/filebeat/current/index.html, also searched existing questions on this forums and thru general Google search. I see that log line offset is shared (per https://www.elastic.co/guide/en/beats/filebeat/current/exported-fields-log.html), but not no. of new log lines which is what we need.