Monitor YARN Application jobs

Elastic Stack Elasticsearch
1

Has anyone done YARN application jobs monitoring with Elastic? If so, how did you do that?

Thanks,
Alex

2

What kind of monitoring are you thinking of doing? Are you looking for ways to detect job/task/container failures? Container log aggregation? Container resource usage? All of the above?

I am willing to bet that some clever usage of Metricbeat and Filebeat installed on each NodeManager could get you pretty far.

3

Thanks James. I want to be able to ship YARN application logs to Logstash and then to Elasticsearch.
Not RM logs or Hadoop metrics, but actual application jobs (Spark, Python, Scala, etc.). And not just the failures but to be able to monitor job progress as well.The challenge is that these logs are stored in HDFS in binary format (TFile), so the requirement is to convert them to text to be able to read (same as "yarn logs..." command does). Hope it clarifies things a bit.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.