How to see if a particular service is logging on a daily basis?

Hello All,

I would like a way to check to see if a group of particular servers are forwarding log data into ElasticSearch on a daily basis.

The current issue is, I am using the nxlog log forwarder to send logs into ElasticSearch on several PaaS instances on Microsoft Azure. However, this has not been going well. If any network issues occur on the Microsoft side, often nxlog is experiencing a loss of connectivity to Logstash, which therefore results in a complete halt of logging on the PaaS instances that experienced the issue.

Since these are production PaaS instances, we cannot just reboot them during the day, and it is not simple to RDP into them because they are PaaS, not IaaS.

We have been racking our brains trying to come up with a way to know when this happens. One person suggested that we utilize the ReST API for ElasticSearch to see if logs are in ElasticSearch on a daily basis for all of these PaaS instances.

I have no idea what kind of query this would be and if it is even possible. So, I am reaching out to you folks for any assistance you could provide.

Thank you for your time,

Michael

Well, you could e.g. make a simple terms aggregation query on whatever field that contains the hostname of each server. Subtract the resulting list of hostnames from the complete list of hosts, and any remaining hosts haven't logged anything during that time period.

If you can get NXLog to ping a separate tool for heartbeat monitoring (e.g. Lovebeat or Riemann) for each message from each host that would be another option.