# Monitor YARN Application jobs

**URL:** <https://discuss.elastic.co/t/monitor-yarn-application-jobs/160443>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [December 11, 2018, 9:44pm UTC](https://discuss.elastic.co/t/monitor-yarn-application-jobs/160443 "2018-12-11T21:44:43Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![amillion8](https://avatars.discourse-cdn.com/v4/letter/a/8c91f0/32.png) [@amillion8](https://discuss.elastic.co/u/amillion8)\
**Post date:** [December 11, 2018, 9:44pm UTC](https://discuss.elastic.co/t/monitor-yarn-application-jobs/160443/1 "2018-12-11T21:44:43Z")

</div>

Has anyone done YARN application jobs monitoring with Elastic? If so, how did you do that?

Thanks,  
Alex

---

<div class="post-metadata">

**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\
**Post date:** [December 12, 2018, 8:31pm UTC](https://discuss.elastic.co/t/monitor-yarn-application-jobs/160443/2 "2018-12-12T20:31:28Z")

</div>

What kind of monitoring are you thinking of doing? Are you looking for ways to detect job/task/container failures? Container log aggregation? Container resource usage? All of the above?

I am willing to bet that some clever usage of Metricbeat and Filebeat installed on each NodeManager could get you pretty far.

---

<div class="post-metadata">

**Author:** ![amillion8](https://avatars.discourse-cdn.com/v4/letter/a/8c91f0/32.png) [@amillion8](https://discuss.elastic.co/u/amillion8)\
**Post date:** [December 12, 2018, 9:47pm UTC](https://discuss.elastic.co/t/monitor-yarn-application-jobs/160443/3 "2018-12-12T21:47:42Z")

</div>

Thanks James. I want to be able to ship YARN application logs to Logstash and then to Elasticsearch.  
Not RM logs or Hadoop metrics, but actual application jobs (Spark, Python, Scala, etc.). And not just the failures but to be able to monitor job progress as well.The challenge is that these logs are stored in HDFS in binary format (TFile), so the requirement is to convert them to text to be able to read (same as "yarn logs..." command does). Hope it clarifies things a bit.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 9, 2019, 9:47pm UTC](https://discuss.elastic.co/t/monitor-yarn-application-jobs/160443/4 "2019-01-09T21:47:46Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.