Checking results of bulk inserts, pre- and post-1.0

axel · April 11, 2014, 2:14pm

Hi,

in my current code, I check "ok": true to see if all my inserts in a bulk
call succeeded.

Now, with 1.0, "ok" is gone. Is there a recommended way to check the
results of individual commands inside a bulk api call? Notably, inserts?

Preferably something that works both pre and post 1.0.

Thanks

Axel

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1475f682-983c-4da8-8cec-7855bc717b69%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

brian_yoder · April 11, 2014, 6:57pm

Axel,

Well, I use the Java API and not the REST API. And while there is a lot
more code around this, here's the core of my bulk load process. I use the
BulkRequestBuilder (a new one for each set of bulk updates!) and not the
BulkProcessor because (a) this code was already written and works and (b)
it gives me some additional control. For instance, I can configure the
maximum error display so that, if I'm loading 1M documents but they all
fail with some stupid fault of mine, I can see the first 128 errors but not
get swamped with all 1 million since they are more than likely to be very
similar.

I first wrote this code for ES 0.19.4 and it has remained unchanged and
working perfectly up to and including ES 1.1.0:

BulkResponse bulkResponse = bulkRequest.execute().actionGet();
if (bulkResponse.hasFailures())
{
if (failedDocuments <= failedMsgLimit)
Sysprint.err.println("BULK WARN: Bulk load failues: "
+ bulkResponse.buildFailureMessage());

BulkItemResponse[] items = bulkResponse.getItems();
for (BulkItemResponse item : items)
{
if (item.isFailed())
{
/* Failure detected /
failedDocuments++;
incrementFailedOpType(item.getOpType());
if (failedDocuments <= failedMsgLimit)
{
Sysprint.err.println("BULK WARN: Bulk failure for "
+ item.getOpType() + ": " + item.getFailureMessage());
}
}
else
{
/ Success: Keep track of version numbers > 1 */
if (item.getVersion() > 1)
setVersionGtOne++;
}
}
}

Hope this helps!

Brian

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7961fd8d-8b7e-431e-bce0-4ac16293b1d3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

axel · April 11, 2014, 11:51pm

Thanks for the response.

My code's in python on top of the rest api, I fail to map the java code
over :-/

Axel

On Friday, April 11, 2014 4:14:10 PM UTC+2, ax...@mozilla.com wrote:

Hi,

in my current code, I check "ok": true to see if all my inserts in a bulk
call succeeded.

Now, with 1.0, "ok" is gone. Is there a recommended way to check the
results of individual commands inside a bulk api call? Notably, inserts?

Preferably something that works both pre and post 1.0.

Thanks

Axel

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/869ecb66-2388-4a42-ac6d-f2adb3befd19%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Honza_Kral · April 12, 2014, 12:03am

Hi axel,

If you are using python you can just use the python client
(elasticsearch-py) it will shield you from this. Just have a look at the
bulk and streaming_bulk helpers in the library.

Hope this helps,
Honza
On Apr 11, 2014 7:52 PM, axel@mozilla.com wrote:

Thanks for the response.

My code's in python on top of the rest api, I fail to map the java code
over :-/

Axel

On Friday, April 11, 2014 4:14:10 PM UTC+2, ax...@mozilla.com wrote:

Hi,

in my current code, I check "ok": true to see if all my inserts in a bulk
call succeeded.

Now, with 1.0, "ok" is gone. Is there a recommended way to check the
results of individual commands inside a bulk api call? Notably, inserts?

Preferably something that works both pre and post 1.0.

Thanks

Axel

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/869ecb66-2388-4a42-ac6d-f2adb3befd19%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/869ecb66-2388-4a42-ac6d-f2adb3befd19%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CABfdDiqyJVmP82F-Hqi3QmTvwCpFjPJ_SsMnJPvUqG5h5YbPMQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

axel · April 23, 2014, 4:43pm

Hi Honza,

sadly, this doesn't seem to work.

Comparing the results from 0.90 vs 1.0.1 from the _bulk api:

{"took":345,"items":[{"index":{"_index":"test","_type":"type1","_id":"1","_version":1,"ok":true}},{"delete":{"_index":"test","_type":"type1","_id":"2","_version":1,"ok":true}},{"create":{"_index":"test","_type":"type1","_id":"3","_version":1,"ok":true}},{"update":{"_index":"index1","_type":"type1","_id":"1","error":"DocumentMissingException[[index1][-1]
[type1][1]: document missing]"}}]}

{"took":256,"errors":true,"items":[{"index":{"_index":"test","_type":"type1","_id":"1","_version":1,"status":201}},{"delete":{"_index":"test","_type":"type1","_id":"2","_version":1,"status":404,"found":false}},{"create":{"_index":"test","_type":"type1","_id":"3","_version":1,"status":201}},{"update":{"_index":"index1","_type":"type1","_id":"1","status":404,"error":"DocumentMissingException[[index1][-1]
[type1][1]: document missing]"}}]}

it seems that pre 1.0 ES doesn't send status, and

github.com

elastic/elasticsearch-py/blob/main/elasticsearch/helpers/init.py#L108


      
          #  Licensed to Elasticsearch B.V. under one or more contributor
          #  license agreements. See the NOTICE file distributed with
          #  this work for additional information regarding copyright
          #  ownership. Elasticsearch B.V. licenses this file to you under
          #  the Apache License, Version 2.0 (the "License"); you may
          #  not use this file except in compliance with the License.
          #  You may obtain a copy of the License at
          #
          # 	http://www.apache.org/licenses/LICENSE-2.0
          #
          #  Unless required by applicable law or agreed to in writing,
          #  software distributed under the License is distributed on an
          #  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
          #  KIND, either express or implied.  See the License for the
          #  specific language governing permissions and limitations
          #  under the License.
          
          from .._async.helpers import async_bulk, async_reindex, async_scan, async_streaming_bulk
          from .._utils import fixup_module_metadata
          from .actions import _chunk_actions  # noqa: F401
          from .actions import _process_bulk_chunk  # noqa: F401
          from .actions import bulk, expand_action, parallel_bulk, reindex, scan, streaming_bulk
          from .errors import BulkIndexError, ScanError
          
          __all__ = [
              "BulkIndexError",
              "ScanError",
              "expand_action",
              "streaming_bulk",
              "bulk",
              "parallel_bulk",
              "scan",
              "reindex",
              "async_scan",
              "async_bulk",
              "async_reindex",
              "async_streaming_bulk",
          ]
          
          fixup_module_metadata(__name__, globals())
          del fixup_module_metadata

backs the lacking status responses up with 500 error codes. So basically
all my inserts fails.

I like the API, though.

RFE: the docs should talk about the return values of APIs, I basically had
to trial and error those.

Axel

Am Samstag, 12. April 2014 02:03:24 UTC+2 schrieb Honza Král:

Hi axel,

If you are using python you can just use the python client
(elasticsearch-py) it will shield you from this. Just have a look at the
bulk and streaming_bulk helpers in the library.

Hope this helps,
Honza
On Apr 11, 2014 7:52 PM, <ax...@mozilla.com <javascript:>> wrote:

Thanks for the response.

My code's in python on top of the rest api, I fail to map the java code
over :-/

Axel

On Friday, April 11, 2014 4:14:10 PM UTC+2, ax...@mozilla.com wrote:

Hi,

in my current code, I check "ok": true to see if all my inserts in a
bulk call succeeded.

Now, with 1.0, "ok" is gone. Is there a recommended way to check the
results of individual commands inside a bulk api call? Notably, inserts?

Preferably something that works both pre and post 1.0.

Thanks

Axel

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/869ecb66-2388-4a42-ac6d-f2adb3befd19%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/869ecb66-2388-4a42-ac6d-f2adb3befd19%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2b3e1512-c9ff-420a-bbc8-97d4d8d078d0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Honza_Kral · April 23, 2014, 10:10pm

Hi Axel,

unfortunately there is no code in python to shield you from the
incompatibilities. There are, however, two releases of
elasticsearch-py - 04.X and 1.0X. Use 0.4.X with elasticsearch 0.90.*
and 1.0.X with elasticsearch 1.*. That should get you what you need.

Honza

On Wed, Apr 23, 2014 at 6:43 PM, axel@mozilla.com wrote:

Hi Honza,

sadly, this doesn't seem to work.

Comparing the results from 0.90 vs 1.0.1 from the _bulk api:

{"took":345,"items":[{"index":{"_index":"test","_type":"type1","_id":"1","_version":1,"ok":true}},{"delete":{"_index":"test","_type":"type1","_id":"2","_version":1,"ok":true}},{"create":{"_index":"test","_type":"type1","_id":"3","_version":1,"ok":true}},{"update":{"_index":"index1","_type":"type1","_id":"1","error":"DocumentMissingException[[index1][-1]
[type1][1]: document missing]"}}]}

{"took":256,"errors":true,"items":[{"index":{"_index":"test","_type":"type1","_id":"1","_version":1,"status":201}},{"delete":{"_index":"test","_type":"type1","_id":"2","_version":1,"status":404,"found":false}},{"create":{"_index":"test","_type":"type1","_id":"3","_version":1,"status":201}},{"update":{"_index":"index1","_type":"type1","_id":"1","status":404,"error":"DocumentMissingException[[index1][-1]
[type1][1]: document missing]"}}]}

it seems that pre 1.0 ES doesn't send status, and
https://github.com/elasticsearch/elasticsearch-py/blob/master/elasticsearch/helpers/__init__.py#L108
backs the lacking status responses up with 500 error codes. So basically all
my inserts fails.

I like the API, though.

RFE: the docs should talk about the return values of APIs, I basically had
to trial and error those.

Axel

Am Samstag, 12. April 2014 02:03:24 UTC+2 schrieb Honza Král:

Hi axel,

If you are using python you can just use the python client
(elasticsearch-py) it will shield you from this. Just have a look at the
bulk and streaming_bulk helpers in the library.

Hope this helps,
Honza

On Apr 11, 2014 7:52 PM, ax...@mozilla.com wrote:

Thanks for the response.

My code's in python on top of the rest api, I fail to map the java code
over :-/

Axel

On Friday, April 11, 2014 4:14:10 PM UTC+2, ax...@mozilla.com wrote:

Hi,

in my current code, I check "ok": true to see if all my inserts in a
bulk call succeeded.

Now, with 1.0, "ok" is gone. Is there a recommended way to check the
results of individual commands inside a bulk api call? Notably, inserts?

Preferably something that works both pre and post 1.0.

Thanks

Axel

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/869ecb66-2388-4a42-ac6d-f2adb3befd19%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/2b3e1512-c9ff-420a-bbc8-97d4d8d078d0%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CABfdDip94aYFBi%3Dnho%3DT17i21AVkYTSgVM43%2BYe5pQ9XTzzrDQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.