According to the doc, it's important to monitor bulk requests to tune ES-Hadoop ingest. I think it will be beneficial to set up monitoring around the requests for production operation as well.
I've tried few different routes to gather the metrics:
- Security audit log and filter successful authentication for bulk request.
- From Kibana's monitoring plugin, and go to index's advanced view, and use Request Rate and Request Time visualization.
- Enable debugging on org.elasticsearch.hadoop.rest.bulk
It looks like option 3 has the richest information, but it's still not enough to answer questions like:
- What's the average bulk request response time?
- How many bulk request is es-hadoop job sending per second?
- How many rejection is happening?
Is there a easier way to gather and consume these metrics?
- Through Kibana's monitoring plugin?
- Expose these metrics as custom Spark metrics?