# How to identify message causing error in bulk request

**URL:** <https://discuss.elastic.co/t/how-to-identify-message-causing-error-in-bulk-request/42885>\
**Category:** Elasticsearch\
**Created:** [February 26, 2016, 6:13pm UTC](https://discuss.elastic.co/t/how-to-identify-message-causing-error-in-bulk-request/42885 "2016-02-26T18:13:32Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![unknownunknown](https://avatars.discourse-cdn.com/v4/letter/u/4da419/32.png) [@unknownunknown](https://discuss.elastic.co/u/unknownunknown)\
**Post date:** [February 26, 2016, 6:13pm UTC](https://discuss.elastic.co/t/how-to-identify-message-causing-error-in-bulk-request/42885/1 "2016-02-26T18:13:32Z")

</div>

I'm using BulkProcessor. Something is causing a problem:

```
[elasticsearch[moo][transport_client_worker][T#19]{New I/O worker #84}] ERROR - Failed to index record: MapperParsingException[failed to parse [_source]]; nested: ElasticsearchParseException[Failed to parse content to map]; nested: JsonParseException[Unexpected character (':' (code 58)): was expecting either valid name character (for unquoted name) or double-quote (for quoted) to start field name#012 at [Source: [B@541daa55; line: 1, column: 1555]];

```

I read this as a JSON parse error in the bulk request. Is that right? How do I turn this into information I can use? I doesn't tell me what index the failure happened in, it doesn't tell me what the message looked like.

```
                  public void afterBulk(long executionId, BulkRequest request, BulkResponse response) {
                            if (response.hasFailures()) {
                                    for (BulkItemResponse item : response.getItems()) {
                                            if (item.isFailed()) {
                                                    logger.error("Failed to index record: " + item.getFailureMessage());
                                            }
                                    }
                            }
                  }

```

Can I get any context about the message that causes the failure in BulkItemResponse, BulkResponse, or BulkRequest? I tried pulling the "payloads" out of the BulkItemResponse, but this didn't seem to correspond to any kind of message body so I can identify where the malformed message is.

Thanks for any help that can be offered.

---

<div class="post-metadata">

**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\
**Post date:** [February 26, 2016, 6:29pm UTC](https://discuss.elastic.co/t/how-to-identify-message-causing-error-in-bulk-request/42885/2 "2016-02-26T18:29:57Z")

</div>

Something like this ought to do:

```java
if (false == response.hasFailures()) {
  return;
}
for (int i = 0; i < response.getItems().size()) {
  if (false == response.getItems().get(i).isFailed()) {
    continue;
  }
  logger.error("Failed to index [" + request.requests().get(i) + "]: [" + response.getItems().get(i).getFailureMessage() + "]");
}

```

Warning: I wrote this inside a little text box on a web page and didn't run it. It is almost certainly wrong. My only goal was to make it obvious that the requests and responses are kept in the same order.

---

<div class="post-metadata">

**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\
**Post date:** [February 26, 2016, 6:31pm UTC](https://discuss.elastic.co/t/how-to-identify-message-causing-error-in-bulk-request/42885/3 "2016-02-26T18:31:03Z")

</div>

Its almost certainly ok to skip the first check - it just iterates and items and looks for one with a failure so this doesn't save any time and it makes the code longer.

---

<div class="post-metadata">

**Author:** ![unknownunknown](https://avatars.discourse-cdn.com/v4/letter/u/4da419/32.png) [@unknownunknown](https://discuss.elastic.co/u/unknownunknown)\
**Post date:** [February 26, 2016, 6:31pm UTC](https://discuss.elastic.co/t/how-to-identify-message-causing-error-in-bulk-request/42885/4 "2016-02-26T18:31:30Z")

</div>

Thanks, I will give this a try and get back to you.

---

<div class="post-metadata">

**Author:** ![unknownunknown](https://avatars.discourse-cdn.com/v4/letter/u/4da419/32.png) [@unknownunknown](https://discuss.elastic.co/u/unknownunknown)\
**Post date:** [February 26, 2016, 6:54pm UTC](https://discuss.elastic.co/t/how-to-identify-message-causing-error-in-bulk-request/42885/5 "2016-02-26T18:54:14Z")

</div>

Thanks! I am hoping this will be very helpful, I got significantly more feedback this way. There were some minor Array vs. ArrayList-isms to get it to work for me, but otherwise not bad for pseudocode.

What ultimately worked for me:

```
                  @Override
                  public void afterBulk(long executionId, BulkRequest request, BulkResponse response) {
                            if (response.hasFailures()) {
                                    for (int i = 0; i < response.getItems().length; i++) {
                                        BulkItemResponse item = response.getItems()[i];
                                        if (item.isFailed()) {
                                              IndexRequest ireq = (IndexRequest) request.requests().get(i);
                                              logger.error("Failed while indexing to " + item.getIndex() + " type " + item.getType() + " " +
                                                           "request: [" + ireq + "]: [" + item.getFailureMessage() + "]");
                                        }
                                    }
                            }
                  }

```

I now get:

[elasticsearch[Dyna-Mite][transport\_client\_worker][T#5]{New I/O worker #70}] ERROR - Failed while indexing to data-2016.02.26 type datatype request: [index {[data-2016.02.26][datatype][null], source[{"json\_obj"}]: [MapperParsingException[failed to parse [\_source]]; nested: ElasticsearchParseException[Failed to parse content to map]; nested: JsonParseException[Unexpected character (':' (code 58)): was expecting either valid name character (for unquoted name) or double-quote (for quoted) to start field name#012 at [Source: [B@5f09799a; line: 1, column: 1779]]; ]

I am hopeful that this will be a great help in troubleshooting this problem, thanks.

---

<div class="post-metadata">

**Author:** ![unknownunknown](https://avatars.discourse-cdn.com/v4/letter/u/4da419/32.png) [@unknownunknown](https://discuss.elastic.co/u/unknownunknown)\
**Post date:** [February 26, 2016, 8:57pm UTC](https://discuss.elastic.co/t/how-to-identify-message-causing-error-in-bulk-request/42885/6 "2016-02-26T20:57:55Z")

</div>

Just wanted to say thanks again. This was a real help for me and allowed me to move forward in my work rather than being frustrated and wondering what was going wrong.

---

<div class="post-metadata">

**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\
**Post date:** [February 26, 2016, 9:13pm UTC](https://discuss.elastic.co/t/how-to-identify-message-causing-error-in-bulk-request/42885/7 "2016-02-26T21:13:59Z")

</div>

Sure! I'm glad I could help!

I am a bit frustrated that the Client interface mixes lists and arrays arbitrarily. We'll get a real java client soon-ish without all the dependencies for Elasticsearch's core and I'll try to do some of the code reviews for it so I can make sure it is consistent about which one it uses.

---

<div class="post-metadata">

**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\
**Post date:** [February 26, 2016, 10:32pm UTC](https://discuss.elastic.co/t/how-to-identify-message-causing-error-in-bulk-request/42885/8 "2016-02-26T22:32:26Z")

</div>

As if working with JSON will make it any easier! 🙂

---

<div class="post-metadata">

**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\
**Post date:** [February 28, 2016, 12:09am UTC](https://discuss.elastic.co/t/how-to-identify-message-causing-error-in-bulk-request/42885/9 "2016-02-28T00:09:41Z")

</div>

I am also curious whether the new Java HTTP client will just throw JSON over the fence, or if it will parse JSON into a new improved ES API.

---

<div class="post-metadata">

**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\
**Post date:** [February 28, 2016, 1:21am UTC](https://discuss.elastic.co/t/how-to-identify-message-causing-error-in-bulk-request/42885/10 "2016-02-28T01:21:08Z")

</div>

I believe their goal is to have a consistent API between all their clients.  
My guess is JSON.

The existing binary API simply throws Map\<String, Object\> over the wire,  
and no one really has written a good ORM for Elasticsearch. Everyone is  
using Jackson databinding or GSON. Since the client will be standalone, I  
wonder if ES will continue using Jackson.

Ivan

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 11:13pm UTC](https://discuss.elastic.co/t/how-to-identify-message-causing-error-in-bulk-request/42885/11 "2017-07-05T23:13:06Z")

</div>