# BulkWriteErrorHandler (6.2) - write failed documents to HDFS

**URL:** <https://discuss.elastic.co/t/bulkwriteerrorhandler-6-2-write-failed-documents-to-hdfs/126282>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [March 30, 2018, 8:43pm UTC](https://discuss.elastic.co/t/bulkwriteerrorhandler-6-2-write-failed-documents-to-hdfs/126282 "2018-03-30T20:43:36Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![dr\_work](https://avatars.discourse-cdn.com/v4/letter/d/4af34b/32.png) [@dr\_work](https://discuss.elastic.co/u/dr_work)\
**Post date:** [March 30, 2018, 8:43pm UTC](https://discuss.elastic.co/t/bulkwriteerrorhandler-6-2-write-failed-documents-to-hdfs/126282/1 "2018-03-30T20:43:37Z")

</div>

In version `6.2`, ErrorHandler functionality was added and it allows to add custom `Bulk Error Handlers`.

In the documentation, the [example](https://www.elastic.co/guide/en/elasticsearch/hadoop/6.2/errorhandlers.html#errorhandlers-bulk-advanced) shows how to write failed documents to a file using `BufferedWriter/OutputStream`.

I need to write the failed documents to `HDFS`. Is it possible to write to HDFS from subclass of BulkWriteErrorHandler?

---

<div class="post-metadata">

**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\
**Post date:** [March 30, 2018, 8:49pm UTC](https://discuss.elastic.co/t/bulkwriteerrorhandler-6-2-write-failed-documents-to-hdfs/126282/2 "2018-03-30T20:49:11Z")

</div>

I don't see why not!

Just an FYI, we are planning on adding more default error handlers as the feature leads up to GA status. An error handler that writes error events to HDFS is one of the ideas we have!

---

<div class="post-metadata">

**Author:** ![dr\_work](https://avatars.discourse-cdn.com/v4/letter/d/4af34b/32.png) [@dr\_work](https://discuss.elastic.co/u/dr_work)\
**Post date:** [March 30, 2018, 9:05pm UTC](https://discuss.elastic.co/t/bulkwriteerrorhandler-6-2-write-failed-documents-to-hdfs/126282/3 "2018-03-30T21:05:30Z")

</div>

Hi James,

I am new to Spark so would appreciate your help. I have been using JavaRDD.saveAsTextFile() to write out text to HDFS.

To convert the document to JavaRDD, I would need access to SparkContext, how can I get access to the SparkContext from my custom `BulkWriteErrorHandler`?

Maybe there is a better way to accomplish this?

Thanks for your help!

---

<div class="post-metadata">

**Author:** ![dr\_work](https://avatars.discourse-cdn.com/v4/letter/d/4af34b/32.png) [@dr\_work](https://discuss.elastic.co/u/dr_work)\
**Post date:** [April 2, 2018, 9:01pm UTC](https://discuss.elastic.co/t/bulkwriteerrorhandler-6-2-write-failed-documents-to-hdfs/126282/4 "2018-04-02T21:01:25Z")

</div>

Hi James,

Could you provide any pointers on how to write to HDFS from BulkWriteErrorHandler? Is it possible to access SparkContext/SparkSession from BulkWriteErrorHandler?

Thanks

---

<div class="post-metadata">

**Author:** ![dr\_work](https://avatars.discourse-cdn.com/v4/letter/d/4af34b/32.png) [@dr\_work](https://discuss.elastic.co/u/dr_work)\
**Post date:** [April 3, 2018, 6:48pm UTC](https://discuss.elastic.co/t/bulkwriteerrorhandler-6-2-write-failed-documents-to-hdfs/126282/5 "2018-04-03T18:48:37Z")

</div>

The following StackOverflow posting helped me out:

> <https://stackoverflow.com/questions/21761285/writing-to-a-file-in-s3-from-jar-on-emr-on-aws>

Thanks.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 1, 2018, 6:48pm UTC](https://discuss.elastic.co/t/bulkwriteerrorhandler-6-2-write-failed-documents-to-hdfs/126282/6 "2018-05-01T18:48:42Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.