Multiple update of one document in one _bulk too slow

Elastic Stack Elasticsearch
1

Hi,

Multiple update of one document in one _bulk is too slow

Example:

curl -XPOST "http://localhost:9200/_bulk" -H 'Content-Type: application/json' -d'
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
'

Elasticsearch version: 5.3, 5.2

Plugins installed:

JVM version: 1.8.0_121-b13

OS version:Ubuntu 16.04.2 LTS

bulk request with 75 updates takes near 6 seconds on our test index that runs on c3.xlarge (4 CPU, 8 GB, documents in inex 14860619, index size 20GB) .
bulk request with 150 updates takes near 12 seconds.

bulk index with 75 documents takes 0.2 seconds

On ES 2.4 - we haven't this problem

Profiler info:

image.png2046×960 252 KB

image.png2042×800 224 KB

If it helps, I may open profiling port to cluster where this problem is reproduced

I create bug on this https://github.com/elastic/elasticsearch/issues/24172 , where advised me to turn off global ordinals.

I turn off global ordinals in mapping for child and reindex all. But it not helps :frowning:
Also I am set refresh_interval to -1. The same result.

2

Just linking my last comment on the issue:

I think the best for you at this point is to rethink how you send documents to elasticsearch.
You should try to lower the number of updates for a single document, you can do it per batch for instance.
Though turning off global ordinals loading on refresh should have helped. Do you still see buildOrdinalMap in your profiler when you disable this option ?

3

Yes

image.png2018×430 109 KB

image.png2066×692 201 KB

mapping

  "stat": {
    "dynamic": "false",
    "_all": {
      "enabled": false
    },
    "_parent": {
      "type": "job",
      "eager_global_ordinals": false
    },
    "_routing": {
      "required": true
    },
    "date_detection": false,
    "properties": {}}

Child documents in index is zero.
I try to reproduce it without childs in mapping

4

Without child relation bulk request with 75 updates takes near 0.5 sec. More than two times slower than index 75 new documents

5

Yes this is expected if you update the same document multiple times in a short period. As I said in 5.x we changed the way we retrieve the documents for an update and if the documents has been updated shortly before the new update this will trigger an automatic refresh of your index. If you have a single document that you update multiple times in your bulk request it will be slower than in 2.x and there is nothing we can do about it. You should find another way to update this document, maybe make a single update after a certain amount of time ?

6

Thanks for answer. Now we use two updates in one bulk for next situation:
Example:
we have doc in ES:

"_source": {
"id": 21129546,
"additional_fields": {
"xxxx": "yyyy",
"zzz": "zzz"
}}

And we need update additional_fields to {"aaa": "vvvvv"}

first update - additional_fields: null
second update additional_fields: {"aaa": "vvvvv"}

Does ES have any mechanism to do it on one request?

I do it using scripting

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.