# Simple Observability of Metrics from Apache Spark
**URL:** https://discuss.elastic.co/t/simple-observability-of-metrics-from-apache-spark/259760
**Category:** Beats
**Tags:** metricbeat
**Created:** [December 28, 2020, 9:31pm UTC](https://discuss.elastic.co/t/simple-observability-of-metrics-from-apache-spark/259760 "2020-12-28T21:31:48Z")
**Posts on this page:** 2
**Page:** 1
### Author:  [@bplies](https://discuss.elastic.co/u/bplies)
#### Post date: [December 28, 2020, 9:31pm UTC](https://discuss.elastic.co/t/simple-observability-of-metrics-from-apache-spark/259760/1 "2020-12-28T21:31:48Z")
We're trying to collect basic metrics from Apache Spark. We realize there are [Hadoop Metrics](https://www.elastic.co/guide/en/elasticsearch/hadoop/current/metrics.html) provided by ES-Hadoop but it seems just so heavy-handed for what we want. We are not trying to tie in full featured interaction between Hadoop and ES, nor track Spark's RDD, nor really use _anything_ listed on ES-Hadoop's [features](https://www.elastic.co/guide/en/elasticsearch/hadoop/current/features.html)
Apache Spark's documentation for [Monitoring and Instrumentation](https://spark.apache.org/docs/2.2.0/monitoring.html) describes how the internal metrics system is based upon Dropwizard (there is an actual Dropwizard Metricbeat [module](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-dropwizard.html), but it doesn't seem to apply here). The documentation describes a number of "Sinks" including one for Graphite but not any one specifically for Elasticsearch.
There is an Elasticsearch sink named [Spelk](https://github.com/ibmruntimes/spelk) out in the wild but it hasn't been maintained in over 4 years, doesn't necessarily format events in an ideal way for Elasticsearch (+ ECS), nor is ILM-aware.
Spark's `MetricsServlet` exposes basic HTTP endpoints for collecting metrics in JSON format, which sounds pretty good if combined with Metricbeat's HTTP [module](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-http.html) which will format events really well and is ILM-aware but the endpoints seem pretty dynamic and not something easily expressed in a static `metricbeat.yml`
What is a good way to go about this? Is there any hope of a simplistic Metricbeat module that can do this? How could the basic HTTP module be reliably used in this situation?
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [January 25, 2021, 11:31pm UTC](https://discuss.elastic.co/t/simple-observability-of-metrics-from-apache-spark/259760/2 "2021-01-25T23:31:51Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.