# Is there a way to determine the timestamp of index consistency **URL:** \ **Category:** Elasticsearch\ **Created:** [November 11, 2021, 10:15pm UTC](https://discuss.elastic.co/t/is-there-a-way-to-determine-the-timestamp-of-index-consistency/289025 "2021-11-11T22:15:34Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Richard\_Ney](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richard_ney/32/79007_2.png) [@Richard\_Ney](https://discuss.elastic.co/u/Richard_Ney)\ **Post date:** [November 11, 2021, 10:15pm UTC](https://discuss.elastic.co/t/is-there-a-way-to-determine-the-timestamp-of-index-consistency/289025/1 "2021-11-11T22:15:34Z")
We're designing a system where we are indexing a large number of document and we want the ability to determine a time watermark where we can claim all documents in an index inserted/update before a certain timestamp are included in a search. Wondering if there is an API call or can we add in a set of "sentinel" documents that we update and then survey at regular interval via search to determine the queryable documents. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [November 12, 2021, 4:13pm UTC](https://discuss.elastic.co/t/is-there-a-way-to-determine-the-timestamp-of-index-consistency/289025/2 "2021-11-12T16:13:04Z")
It sounds like you want the [refresh API](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-refresh.html). If a refresh is successful then all documents acknowledged before the refresh API was called will be visible to searches. ---
**Author:** ![Richard\_Ney](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richard_ney/32/79007_2.png) [@Richard\_Ney](https://discuss.elastic.co/u/Richard_Ney)\ **Post date:** [November 12, 2021, 5:41pm UTC](https://discuss.elastic.co/t/is-there-a-way-to-determine-the-timestamp-of-index-consistency/289025/3 "2021-11-12T17:41:36Z")
Based on observations of my use of the refresh API this wouldn't scale. Our distributed application is ingesting 1000s of updates a second across multiple tenants and in the event a change is made to policy we need to execute a textual search of the prior ingested documents to find exceptions to the rules while applying those rules to the ingestion feed. We're trying to close the window between the enactment of a new policy and the execution of a search against prior indexed data to catch all documents ingested before the policy was applied. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [November 12, 2021, 6:31pm UTC](https://discuss.elastic.co/t/is-there-a-way-to-determine-the-timestamp-of-index-consistency/289025/4 "2021-11-12T18:31:53Z")
Doesn't scale in what sense? How often are you changing these policies? You'd only need to refresh at each policy change AIUI. ---
**Author:** ![Richard\_Ney](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richard_ney/32/79007_2.png) [@Richard\_Ney](https://discuss.elastic.co/u/Richard_Ney)\ **Post date:** [November 12, 2021, 9:03pm UTC](https://discuss.elastic.co/t/is-there-a-way-to-determine-the-timestamp-of-index-consistency/289025/5 "2021-11-12T21:03:58Z")
The policy changes are controlled by our customers so it varies from a couple 100 times a day to 1,000s of times a day all while the infrastructure is ingesting millions of document updates a day and as our customer base grows the number of policy changes grow and our regularly running background queries grow. Now I'll admit the index I use the \_refresh API on the most has a 60 sec refresh rate since it has fairly large documents/heavily nested and mainly drives a UI and when I call \_refresh it normally takes a couple seconds to return. The indexes I'm trying to ensure consistency on have refresh rates of 500ms - 1s, is the expectation in that case that a call to \_refresh returns in milliseconds? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [November 12, 2021, 10:21pm UTC](https://discuss.elastic.co/t/is-there-a-way-to-determine-the-timestamp-of-index-consistency/289025/6 "2021-11-12T22:21:37Z")
I don't think I'm understanding the problem. 1000 times per day is not quite once per minute, there should be no problem refreshing at that sort of frequency. I also don't follow why it's a problem if it takes a couple of seconds for the refresh to complete. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 10, 2021, 10:22pm UTC](https://discuss.elastic.co/t/is-there-a-way-to-determine-the-timestamp-of-index-consistency/289025/7 "2021-12-10T22:22:08Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.