# Add Es Spark Accumulators
**URL:** https://discuss.elastic.co/t/add-es-spark-accumulators/347127
**Category:** Elasticsearch
**Tags:** es-hadoop
**Created:** [November 14, 2023, 2:03pm UTC](https://discuss.elastic.co/t/add-es-spark-accumulators/347127 "2023-11-14T14:03:44Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@glegoux](https://discuss.elastic.co/u/glegoux)
#### Post date: [November 14, 2023, 2:03pm UTC](https://discuss.elastic.co/t/add-es-spark-accumulators/347127/1 "2023-11-14T14:03:44Z")
Hello 👋,
I did a [pull request](https://github.com/elastic/elasticsearch-hadoop/pull/2159).
These metrics will be very useful to monitor a Spark application using the extension Elasticsearch for Hadoop.
What do you think about it?
---
### Author:  [@Keith\_Massey](https://discuss.elastic.co/u/Keith_Massey)
#### Post date: [November 21, 2023, 1:57pm UTC](https://discuss.elastic.co/t/add-es-spark-accumulators/347127/2 "2023-11-21T13:57:15Z")
Thanks for the PR! That seems like a good idea, assuming you're not updating the accumulators inside of a transformation (I haven't looked at the code yet) since that could double-count things. The whole team has been incredibly busy the last few weeks, but we definitely plan on taking a look at this.
---
### Author:  [@glegoux](https://discuss.elastic.co/u/glegoux)
#### Post date: [November 21, 2023, 9:30pm UTC](https://discuss.elastic.co/t/add-es-spark-accumulators/347127/3 "2023-11-21T21:30:41Z")
Happy 😊 to know that this feature seems also useful to the core team!
Yes, with the lazy behavior of Spark, the attempts of Spark tasks and the retries of embedded Elasticsearch client provoke that the counters can be increased several times for the same batch or Spark task. But it makes sense for each metric because you monitor the interactions' client / server, where the communications that can be redundant to export / import a same portion of data, or where the code is executed several times because the RDD/DataFrame/Dataset is not persisted in cache.
The export from Elasticsearch (reads) is a Spark transformation, and the import to Elasticsearch (writes) is a Spark action.
I wait for with haste 🤩 your code review and feedbacks to adapt and improve the code!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 19, 2023, 9:30pm UTC](https://discuss.elastic.co/t/add-es-spark-accumulators/347127/4 "2023-12-19T21:30:49Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.