Refreshes happening due to the Update API

Hey,

I have found a case where an unintended refresh happens, when you try to update a document, that has not been refreshed yet. Take the following example:

DELETE test 

PUT test
{"settings":{"refresh_interval":"1h"}}

PUT test/_doc/1?refresh
{"foo":"bar"}

# refresh happened
GET test/_stats?human&filter_path=_all.primaries.refresh.total

POST test/_update/1?refresh=false
{"doc":{"foo":"bar1"}}

# no refresh should happen
GET test/_stats?human&filter_path=_all.primaries.refresh.total

POST test/_update/1?refresh=false
{"doc":{"foo":"bar2"}}

# OHAI REFRESH - where does it come from?
GET test/_stats?human&filter_path=_all.primaries.refresh.total

Running this on a current 8.x release returns a refresh after the last update despite explicitely setting refresh=false.

Can someone confirm/deny that this refresh is happening as intended?

We do have a lot of updates that probably fall into this case due to different consumers writing to Elasticsearch at roughly the same time, so that this may be triggered quite often.

Any suggestions to reduce these ongoing refreshes without synchronizing before writing to Elasticsearch?

--Alex

1 Like

Updating myself here a little...

according to the source there is the possibility of a refresh happening when running gets, see elasticsearch/server/src/main/java/org/elasticsearch/index/engine/InternalEngine.java at 7.17 · elastic/elasticsearch · GitHub

Still trying to figure out, if that can be prevented/reduced, so any further help appreciated.

2 Likes

This is kinda subtle. The _update API does a realtime get to obtain the latest version of the doc, followed by a write with optimistic concurrency control to ensure that there were no concurrent writes. The refresh you're seeing is coming from the realtime get.

Realtime gets may read recent operations from the translog, but can only do this if the location of the document in the translog is tracked in memory. This tracking is itself expensive, and it's unnecessary unless you're doing realtime gets, so it's disabled until the shard sees a realtime get happening. If the translog location is unavailable then a realtime get must perform a refresh so it can retrieve the document with a search instead.

Thus we'd expect the first update on a shard to trigger a refresh, as you've observed, because at this point the shard isn't tracking translog locations in memory. The first update also flips it into tracking-translog-locations mode so that subsequent updates will be able to read docs directly from the translog without needing further refreshes, and indeed that's what we can observe:

DELETE /test

# 200 OK
# {
#   "acknowledged": true
# }

PUT /test
{
  "settings": {
    "number_of_replicas": 0,
    "number_of_shards": 1,
    "refresh_interval": -1
  }
}

# 200 OK
# {
#   "acknowledged": true,
#   "index": "test",
#   "shards_acknowledged": true
# }

NB creating a single shard, no replicas, otherwise we need to flip each shard copy into tracking-translog-locations mode which requires N gets and therefore N refreshes.

PUT /test/_doc/1?refresh
{
  "foo": "bar"
}

# 201 Created
# {
#   "_id": "1",
#   "_index": "test",
#   "_primary_term": 1,
#   "_seq_no": 0,
#   "_shards": {
#     "failed": 0,
#     "successful": 1,
#     "total": 1
#   },
#   "_version": 1,
#   "forced_refresh": true,
#   "result": "created"
# }

GET /test/_stats?human&filter_path=_all.primaries.refresh.total

# 200 OK
# {
#   "_all": {
#     "primaries": {
#       "refresh": {
#         "total": 3
#       }
#     }
#   }
# }

POST /test/_update/1?refresh=false
{
  "doc": {
    "foo": "bar1"
  }
}

# 200 OK
# {
#   "_id": "1",
#   "_index": "test",
#   "_primary_term": 1,
#   "_seq_no": 1,
#   "_shards": {
#     "failed": 0,
#     "successful": 1,
#     "total": 1
#   },
#   "_version": 2,
#   "result": "updated"
# }

GET /test/_stats?human&filter_path=_all.primaries.refresh.total

# 200 OK
# {
#   "_all": {
#     "primaries": {
#       "refresh": {
#         "total": 3
#       }
#     }
#   }
# }

NB no refresh was needed for this update, the index was already fully refreshed.

POST /test/_update/1?refresh=false
{
  "doc": {
    "foo": "bar2"
  }
}

# 200 OK
# {
#   "_id": "1",
#   "_index": "test",
#   "_primary_term": 1,
#   "_seq_no": 2,
#   "_shards": {
#     "failed": 0,
#     "successful": 1,
#     "total": 1
#   },
#   "_version": 3,
#   "result": "updated"
# }

GET /test/_stats?human&filter_path=_all.primaries.refresh.total

# 200 OK
# {
#   "_all": {
#     "primaries": {
#       "refresh": {
#         "total": 4
#       }
#     }
#   }
# }

NB a refresh was needed for this update because the shard was not in translog-location-tracking mode so had to get the previous document update using a search. But now the shard is in translog-location-tracking mode ...

POST /test/_update/1?refresh=false
{
  "doc": {
    "foo": "bar3"
  }
}

# 200 OK
# {
#   "_id": "1",
#   "_index": "test",
#   "_primary_term": 1,
#   "_seq_no": 3,
#   "_shards": {
#     "failed": 0,
#     "successful": 1,
#     "total": 1
#   },
#   "_version": 4,
#   "result": "updated"
# }

GET /test/_stats?human&filter_path=_all.primaries.refresh.total

# 200 OK
# {
#   "_all": {
#     "primaries": {
#       "refresh": {
#         "total": 4
#       }
#     }
#   }
# }

... so this update needs no refresh ...

POST /test/_update/1?refresh=false
{
  "doc": {
    "foo": "bar4"
  }
}

# 200 OK
# {
#   "_id": "1",
#   "_index": "test",
#   "_primary_term": 1,
#   "_seq_no": 4,
#   "_shards": {
#     "failed": 0,
#     "successful": 1,
#     "total": 1
#   },
#   "_version": 5,
#   "result": "updated"
# }

GET /test/_stats?human&filter_path=_all.primaries.refresh.total

# 200 OK
# {
#   "_all": {
#     "primaries": {
#       "refresh": {
#         "total": 4
#       }
#     }
#   }
# }

... and nor does this one.

1 Like

That indeed explains the behavior, thanks David!

Does this behavior change again with many document updates (hundreds, sometimes several thousands per minute), which results in constant refreshes and segment creations (and follow up merges)?

Also you said it is memory intensive. is that configurable somewhere, as we could use some more heap to reduce the problem.

Is this behavior somewhere documented? I think it's not too uncommon nowadays to have async updates on different parts of documents via consumers like Kafka, that make this more likely to hit as a user and I feel it makes sense to know about it.

I don't think so, no. I mean if you're doing a lot of writes then we'll be creating segments more quickly just to keep the indexing buffer size under control. But I wouldn't expect to see more refreshes related to realtime gets.

No (except in the source ofc). It's very much an implementation detail, and not something on which we'd want users to rely.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.