# Checking results of bulk inserts, pre- and post-1.0 **URL:** \ **Category:** Elasticsearch\ **Created:** [April 11, 2014, 2:14pm UTC](https://discuss.elastic.co/t/checking-results-of-bulk-inserts-pre-and-post-1-0/16960 "2014-04-11T14:14:10Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![axel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axel/32/1620_2.png) [@axel](https://discuss.elastic.co/u/axel)\ **Post date:** [April 11, 2014, 2:14pm UTC](https://discuss.elastic.co/t/checking-results-of-bulk-inserts-pre-and-post-1-0/16960/1 "2014-04-11T14:14:10Z")
Hi, in my current code, I check "ok": true to see if all my inserts in a bulk call succeeded. Now, with 1.0, "ok" is gone. Is there a recommended way to check the results of individual commands inside a bulk api call? Notably, inserts? Preferably something that works both pre and post 1.0. Thanks Axel -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1475f682-983c-4da8-8cec-7855bc717b69%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1475f682-983c-4da8-8cec-7855bc717b69%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![brian\_yoder](https://avatars.discourse-cdn.com/v4/letter/b/f1d935/32.png) [@brian\_yoder](https://discuss.elastic.co/u/brian_yoder)\ **Post date:** [April 11, 2014, 6:57pm UTC](https://discuss.elastic.co/t/checking-results-of-bulk-inserts-pre-and-post-1-0/16960/2 "2014-04-11T18:57:06Z")
Axel, Well, I use the Java API and not the REST API. And while there is a lot more code around this, here's the core of my bulk load process. I use the BulkRequestBuilder (a new one for each set of bulk updates!) and not the BulkProcessor because (a) this code was already written and works and (b) it gives me some additional control. For instance, I can configure the maximum error display so that, if I'm loading 1M documents but they all fail with some stupid fault of mine, I can see the first 128 errors but not get swamped with all 1 million since they are more than likely to be very similar. I first wrote this code for ES 0.19.4 and it has remained unchanged and working perfectly up to and including ES 1.1.0: BulkResponse bulkResponse = bulkRequest.execute().actionGet(); if (bulkResponse.hasFailures()) { if (failedDocuments \<= failedMsgLimit) Sysprint.err.println("BULK WARN: Bulk load failues: " + bulkResponse.buildFailureMessage()); BulkItemResponse[] items = bulkResponse.getItems(); for (BulkItemResponse item : items) { if (item.isFailed()) { /\* Failure detected _/ failedDocuments++; incrementFailedOpType(item.getOpType()); if (failedDocuments \<= failedMsgLimit) { Sysprint.err.println("BULK WARN: Bulk failure for " + item.getOpType() + ": " + item.getFailureMessage()); } } else { /_ Success: Keep track of version numbers \> 1 \*/ if (item.getVersion() \> 1) setVersionGtOne++; } } } Hope this helps! Brian -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/7961fd8d-8b7e-431e-bce0-4ac16293b1d3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7961fd8d-8b7e-431e-bce0-4ac16293b1d3%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![axel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axel/32/1620_2.png) [@axel](https://discuss.elastic.co/u/axel)\ **Post date:** [April 11, 2014, 11:51pm UTC](https://discuss.elastic.co/t/checking-results-of-bulk-inserts-pre-and-post-1-0/16960/3 "2014-04-11T23:51:57Z")
Thanks for the response. My code's in python on top of the rest api, I fail to map the java code over :-/ Axel On Friday, April 11, 2014 4:14:10 PM UTC+2, [ax...@mozilla.com](mailto:ax...@mozilla.com) wrote: > Hi, > > in my current code, I check "ok": true to see if all my inserts in a bulk > call succeeded. > > Now, with 1.0, "ok" is gone. Is there a recommended way to check the > results of individual commands inside a bulk api call? Notably, inserts? > > Preferably something that works both pre and post 1.0. > > Thanks > > Axel -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/869ecb66-2388-4a42-ac6d-f2adb3befd19%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/869ecb66-2388-4a42-ac6d-f2adb3befd19%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Honza\_Kral](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/honza_kral/32/876_2.png) [@Honza\_Kral](https://discuss.elastic.co/u/Honza_Kral)\ **Post date:** [April 12, 2014, 12:03am UTC](https://discuss.elastic.co/t/checking-results-of-bulk-inserts-pre-and-post-1-0/16960/4 "2014-04-12T00:03:24Z")
Hi axel, If you are using python you can just use the python client (elasticsearch-py) it will shield you from this. Just have a look at the bulk and streaming\_bulk helpers in the library. Hope this helps, Honza On Apr 11, 2014 7:52 PM, [axel@mozilla.com](mailto:axel@mozilla.com) wrote: > Thanks for the response. > > My code's in python on top of the rest api, I fail to map the java code > over :-/ > > Axel > > On Friday, April 11, 2014 4:14:10 PM UTC+2, [ax...@mozilla.com](mailto:ax...@mozilla.com) wrote: > > > Hi, > > > > in my current code, I check "ok": true to see if all my inserts in a bulk > > call succeeded. > > > > Now, with 1.0, "ok" is gone. Is there a recommended way to check the > > results of individual commands inside a bulk api call? Notably, inserts? > > > > Preferably something that works both pre and post 1.0. > > > > Thanks > > > > Axel > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/869ecb66-2388-4a42-ac6d-f2adb3befd19%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/869ecb66-2388-4a42-ac6d-f2adb3befd19%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/869ecb66-2388-4a42-ac6d-f2adb3befd19%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/869ecb66-2388-4a42-ac6d-f2adb3befd19%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CABfdDiqyJVmP82F-Hqi3QmTvwCpFjPJ\_SsMnJPvUqG5h5YbPMQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CABfdDiqyJVmP82F-Hqi3QmTvwCpFjPJ_SsMnJPvUqG5h5YbPMQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![axel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axel/32/1620_2.png) [@axel](https://discuss.elastic.co/u/axel)\ **Post date:** [April 23, 2014, 4:43pm UTC](https://discuss.elastic.co/t/checking-results-of-bulk-inserts-pre-and-post-1-0/16960/5 "2014-04-23T16:43:16Z")
Hi Honza, sadly, this doesn't seem to work. Comparing the results from 0.90 vs 1.0.1 from the \_bulk api: {"took":345,"items":[{"index":{"\_index":"test","\_type":"type1","\_id":"1","\_version":1,"ok":true}},{"delete":{"\_index":"test","\_type":"type1","\_id":"2","\_version":1,"ok":true}},{"create":{"\_index":"test","\_type":"type1","\_id":"3","\_version":1,"ok":true}},{"update":{"\_index":"index1","\_type":"type1","\_id":"1","error":"DocumentMissingException[[index1][-1] [type1][1]: document missing]"}}]} {"took":256,"errors":true,"items":[{"index":{"\_index":"test","\_type":"type1","\_id":"1","\_version":1,"status":201}},{"delete":{"\_index":"test","\_type":"type1","\_id":"2","\_version":1,"status":404,"found":false}},{"create":{"\_index":"test","\_type":"type1","\_id":"3","\_version":1,"status":201}},{"update":{"\_index":"index1","\_type":"type1","\_id":"1","status":404,"error":"DocumentMissingException[[index1][-1] [type1][1]: document missing]"}}]} it seems that pre 1.0 ES doesn't send status, and > backs the lacking status responses up with 500 error codes. So basically all my inserts fails. I like the API, though. RFE: the docs should talk about the return values of APIs, I basically had to trial and error those. Axel Am Samstag, 12. April 2014 02:03:24 UTC+2 schrieb Honza Král: > Hi axel, > > If you are using python you can just use the python client > (elasticsearch-py) it will shield you from this. Just have a look at the > bulk and streaming\_bulk helpers in the library. > > Hope this helps, > Honza > On Apr 11, 2014 7:52 PM, \<[ax...@mozilla.com](mailto:ax...@mozilla.com) \\> wrote: > > > Thanks for the response. > > > > My code's in python on top of the rest api, I fail to map the java code > > over :-/ > > > > Axel > > > > On Friday, April 11, 2014 4:14:10 PM UTC+2, [ax...@mozilla.com](mailto:ax...@mozilla.com) wrote: > > > > > Hi, > > > > > > in my current code, I check "ok": true to see if all my inserts in a > > > bulk call succeeded. > > > > > > Now, with 1.0, "ok" is gone. Is there a recommended way to check the > > > results of individual commands inside a bulk api call? Notably, inserts? > > > > > > Preferably something that works both pre and post 1.0. > > > > > > Thanks > > > > > > Axel > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/869ecb66-2388-4a42-ac6d-f2adb3befd19%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/869ecb66-2388-4a42-ac6d-f2adb3befd19%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/869ecb66-2388-4a42-ac6d-f2adb3befd19%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/869ecb66-2388-4a42-ac6d-f2adb3befd19%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/2b3e1512-c9ff-420a-bbc8-97d4d8d078d0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/2b3e1512-c9ff-420a-bbc8-97d4d8d078d0%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Honza\_Kral](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/honza_kral/32/876_2.png) [@Honza\_Kral](https://discuss.elastic.co/u/Honza_Kral)\ **Post date:** [April 23, 2014, 10:10pm UTC](https://discuss.elastic.co/t/checking-results-of-bulk-inserts-pre-and-post-1-0/16960/6 "2014-04-23T22:10:58Z")
Hi Axel, unfortunately there is no code in python to shield you from the incompatibilities. There are, however, two releases of elasticsearch-py - 04.X and 1.0X. Use 0.4.X with elasticsearch 0.90.\* and 1.0.X with elasticsearch 1.\*. That should get you what you need. Honza On Wed, Apr 23, 2014 at 6:43 PM, [axel@mozilla.com](mailto:axel@mozilla.com) wrote: > Hi Honza, > > sadly, this doesn't seem to work. > > Comparing the results from 0.90 vs 1.0.1 from the \_bulk api: > > {"took":345,"items":[{"index":{"\_index":"test","\_type":"type1","\_id":"1","\_version":1,"ok":true}},{"delete":{"\_index":"test","\_type":"type1","\_id":"2","\_version":1,"ok":true}},{"create":{"\_index":"test","\_type":"type1","\_id":"3","\_version":1,"ok":true}},{"update":{"\_index":"index1","\_type":"type1","\_id":"1","error":"DocumentMissingException[[index1][-1] > [type1][1]: document missing]"}}]} > > {"took":256,"errors":true,"items":[{"index":{"\_index":"test","\_type":"type1","\_id":"1","\_version":1,"status":201}},{"delete":{"\_index":"test","\_type":"type1","\_id":"2","\_version":1,"status":404,"found":false}},{"create":{"\_index":"test","\_type":"type1","\_id":"3","\_version":1,"status":201}},{"update":{"\_index":"index1","\_type":"type1","\_id":"1","status":404,"error":"DocumentMissingException[[index1][-1] > [type1][1]: document missing]"}}]} > > it seems that pre 1.0 ES doesn't send status, and > [https://github.com/elasticsearch/elasticsearch-py/blob/master/elasticsearch/helpers/\_\_init\_\_.py#L108](https://github.com/elasticsearch/elasticsearch-py/blob/master/elasticsearch/helpers/ __init__.py#L108) > backs the lacking status responses up with 500 error codes. So basically all > my inserts fails. > > I like the API, though. > > RFE: the docs should talk about the return values of APIs, I basically had > to trial and error those. > > Axel > > Am Samstag, 12. April 2014 02:03:24 UTC+2 schrieb Honza Král: > > > Hi axel, > > > > If you are using python you can just use the python client > > (elasticsearch-py) it will shield you from this. Just have a look at the > > bulk and streaming\_bulk helpers in the library. > > > > Hope this helps, > > Honza > > > > On Apr 11, 2014 7:52 PM, [ax...@mozilla.com](mailto:ax...@mozilla.com) wrote: > > > > > Thanks for the response. > > > > > > My code's in python on top of the rest api, I fail to map the java code > > > over :-/ > > > > > > Axel > > > > > > On Friday, April 11, 2014 4:14:10 PM UTC+2, [ax...@mozilla.com](mailto:ax...@mozilla.com) wrote: > > > > > > > Hi, > > > > > > > > in my current code, I check "ok": true to see if all my inserts in a > > > > bulk call succeeded. > > > > > > > > Now, with 1.0, "ok" is gone. Is there a recommended way to check the > > > > results of individual commands inside a bulk api call? Notably, inserts? > > > > > > > > Preferably something that works both pre and post 1.0. > > > > > > > > Thanks > > > > > > > > Axel > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/869ecb66-2388-4a42-ac6d-f2adb3befd19%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/869ecb66-2388-4a42-ac6d-f2adb3befd19%40googlegroups.com). > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/2b3e1512-c9ff-420a-bbc8-97d4d8d078d0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/2b3e1512-c9ff-420a-bbc8-97d4d8d078d0%40googlegroups.com). > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CABfdDip94aYFBi%3Dnho%3DT17i21AVkYTSgVM43%2BYe5pQ9XTzzrDQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CABfdDip94aYFBi%3Dnho%3DT17i21AVkYTSgVM43%2BYe5pQ9XTzzrDQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:33am UTC](https://discuss.elastic.co/t/checking-results-of-bulk-inserts-pre-and-post-1-0/16960/7 "2017-07-06T01:33:55Z")