# Python script update by query elasticsearch doesn't work

**URL:** <https://discuss.elastic.co/t/python-script-update-by-query-elasticsearch-doesnt-work/199334>\
**Category:** Elasticsearch\
**Created:** [September 12, 2019, 10:40pm UTC](https://discuss.elastic.co/t/python-script-update-by-query-elasticsearch-doesnt-work/199334 "2019-09-12T22:40:22Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![EZprogramming](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ezprogramming/32/57291_2.png) [@EZprogramming](https://discuss.elastic.co/u/EZprogramming)\
**Post date:** [September 12, 2019, 10:40pm UTC](https://discuss.elastic.co/t/python-script-update-by-query-elasticsearch-doesnt-work/199334/1 "2019-09-12T22:40:22Z")

</div>

My python script fails at performing the following operation which is what I found in here:  
[[https://stackoverflow.com/questions/42489340/elastisearch-update-by-query](https://stackoverflow.com/questions/42489340/elastisearch-update-by-query)]

```
def execute_es_update_query(self, i):
    t1 = {
        "script": {
            "inline": "ctx._source.tomato_activity = 'Yes'"
        },
        "query": {
            "range":{
                "tomato_count": {
                    "gte": 0
                }
            }
        }
    }

    t2 = {
        "script": {
            "inline": "ctx._source.tomato_activity = 'No'"
        },
        "query": {
            "match":{
                "tomato_count": 0
            }
        }
    }

    t3 = {
        "script": {
            "inline": "ctx._source.tomato_activity = 'None'"
        },
        "query": {
            "range":{
                "tomato_count": {
                    "lte": 0
                }
            }
        }
    }

    r1 = {
        "script": {
            "inline": "ctx._source.row_activity = 'Yes'"
        },
        "query": {
            "range":{
                "row_captured": {
                    "gte": 1
                }
            }
        }
    }

    r2 = {
        "script": {
            "inline": "ctx._source.row_activity = 'No'"
        },
        "query": {
            "match":{
                "row_captured": 0
            }
        }
    }

    r3 = {
        "script": {
            "inline": "ctx._source.row_activity = 'None'"
        },
        "query": {
            "range":{
                "row_captured": {
                    "lte": 0
                }
            }
        }
    }
    
    queries = list()
    queries = [t1, t2, t3, r1, r2, r3]

    if self.es.indices.exists(index = i):
        for q in queries:
            self.es.update_by_query(index = i, body = q)

```

**Error:**

```
Traceback (most recent call last):
  File "/usr/share/logstash/script/customer_model/run.py", line 19, in <module>
    ecoation.execute_operations()
  File "/usr/share/logstash/script/customer_model/merge.py", line 262, in execute_operations
    self.execute_es_update_query(self.index)
  File "/usr/share/logstash/script/customer_model/merge.py", line 224, in execute_es_update_query
    self.es.update_by_query(index = i, body = q)
  File "/usr/local/lib/python3.6/dist-packages/elasticsearch/client/utils.py", line 84, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/elasticsearch/client/ __init__.py", line 942, in update_by_query
    "POST", _make_path(index, "_update_by_query"), params=params, body=body
  File "/usr/local/lib/python3.6/dist-packages/elasticsearch/transport.py", line 350, in perform_request
    timeout=timeout,
  File "/usr/local/lib/python3.6/dist-packages/elasticsearch/connection/http_urllib3.py", line 252, in perform_request
    self._raise_error(response.status, raw_data)
  File "/usr/local/lib/python3.6/dist-packages/elasticsearch/connection/base.py", line 181, in _raise_error
    status_code, error_message, additional_info
elasticsearch.exceptions.ConflictError: ConflictError(409, '{"took":259,"timed_out":false,"total":132,"updated":0,"deleted":0,"batches":1,"version_conflicts":132,"noops":0,"retries":{"bulk":0,"search":0},"throttled_millis":0,"requests_per_second":-1.0,"throttled_until_millis":0,"failures":[{"index":"api-athena-fruitcount-test-2019.09.12","type":"_doc","id":"dO1lJ20BLWZVc3oBj40F","cause":{"type":"version_conflict_engine_exception","reason":"[dO1lJ20BLWZVc3oBj40F]: version conflict, required seqNo [0], primary term [1]. current document has seqNo [192] and primary term [1]","index_uuid":"EwJeXIC1RyehtGOwKIHArw","shard":"0","index":"api-athena-fruitcount-test-2019.09.12"},"status":409},
```

---

<div class="post-metadata">

**Author:** ![EZprogramming](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ezprogramming/32/57291_2.png) [@EZprogramming](https://discuss.elastic.co/u/EZprogramming)\
**Post date:** [September 12, 2019, 11:06pm UTC](https://discuss.elastic.co/t/python-script-update-by-query-elasticsearch-doesnt-work/199334/2 "2019-09-12T23:06:47Z")

</div>

**Solution:**

**Reference:**

> [@Elasticsearch delete\_by\_query 409 version conflict](https://discuss.elastic.co/t/elasticsearch-delete-by-query-409-version-conflict/174150/3):
>
> Hi @HenningAndersen, So \_delete\_by\_query basically searches for the documents to delete and then deletes them one by one. And a version conflict occurs if one or more of the documents gets update in between the time when the search was completed and the delete operation was started. But as I said, I had received a successful created/updated response for all the documents that have to deleted, before sending the \_delete\_by\_query request. And I am pretty sure that that none of the documents are …

I made this change to my python code and it worked:

I added `self.es.indices.refresh(index = i)`

```
if self.es.indices.exists(index = i):
            for q in queries:
                self.es.indices.refresh(index = i)
                self.es.update_by_query(index = i, body = q)

```

---

<div class="post-metadata">

**Author:** ![gabriel\_tessier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gabriel_tessier/32/27911_2.png) [@gabriel\_tessier](https://discuss.elastic.co/u/gabriel_tessier)\
**Post date:** [September 13, 2019, 12:42am UTC](https://discuss.elastic.co/t/python-script-update-by-query-elasticsearch-doesnt-work/199334/3 "2019-09-13T00:42:21Z")

</div>

Hi @EZprogramming

Just one information about your solution to refresh the index before each query, you need to know about the cost of a refresh call.

> <https://stackoverflow.com/questions/19963406/refresh-vs-flush>

  

> **[Elasticsearch from the Bottom Up, Part 1](https://www.elastic.co/blog/found-elasticsearch-from-the-bottom-up)**
>
> In this article series, we look at Elasticsearch from a new perspective. We'll start at the 'bottom' (or close enough!) of the many abstraction levels, and gradually move upwards towards the user-visible layers, studying the various internal data...

  
[http://blog.mikemccandless.com/](http://blog.mikemccandless.com/) \<--- this blog a little bit old but have a lot of resources about how lucene work.

Knowing that and to prevent run unnecessary refresh can be keep the code as before and add a try catch if the exception is a `ConflictError` so you can run a refresh and call again your update.  
something like:

```auto
    from elasticsearch.exceptions import ConflictError

    if self.es.indices.exists(index = i):
        for q in queries:
            try:
                self.es.update_by_query(index = i, body = q)
            except ConflictError as e:
                # a conflict exception is raised more info here: https://discuss.elastic.co/t/python-script-update-by-query-elasticsearch-doesnt-work
                self.es.indices.refresh(index = i)
                # try again
                self.es.update_by_query(index = i, body = q)

```

---

<div class="post-metadata">

**Author:** ![EZprogramming](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ezprogramming/32/57291_2.png) [@EZprogramming](https://discuss.elastic.co/u/EZprogramming)\
**Post date:** [September 13, 2019, 12:45am UTC](https://discuss.elastic.co/t/python-script-update-by-query-elasticsearch-doesnt-work/199334/4 "2019-09-13T00:45:36Z")

</div>

@gabriel_tessier, thanks for pointing that out. Your approach is more efficient, I agree.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 11, 2019, 12:45am UTC](https://discuss.elastic.co/t/python-script-update-by-query-elasticsearch-doesnt-work/199334/5 "2019-10-11T00:45:42Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.