# Doing periodic searches via API

**URL:** <https://discuss.elastic.co/t/doing-periodic-searches-via-api/269268>\
**Category:** Elasticsearch\
**Created:** [April 5, 2021, 8:36pm UTC](https://discuss.elastic.co/t/doing-periodic-searches-via-api/269268 "2021-04-05T20:36:47Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Russell\_Fulton](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/russell_fulton/32/62888_2.png) [@Russell\_Fulton](https://discuss.elastic.co/u/Russell_Fulton)\
**Post date:** [April 5, 2021, 8:36pm UTC](https://discuss.elastic.co/t/doing-periodic-searches-via-api/269268/1 "2021-04-05T20:36:47Z")

</div>

I have an application where I need to run searches on an index periodically picking up "new" entries from the index and I am wondering the best way of doing this.

I assume that \_id is serial and simply saving the \_id of the last document in each run and doing the next search for \_id greater than my saved value should work. Correct ?

Is there a better way to do this?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 5, 2021, 9:25pm UTC](https://discuss.elastic.co/t/doing-periodic-searches-via-api/269268/2 "2021-04-05T21:25:07Z")

</div>

What about adding a timestamp to each document and select documents added after a given date?

---

<div class="post-metadata">

**Author:** ![Russell\_Fulton](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/russell_fulton/32/62888_2.png) [@Russell\_Fulton](https://discuss.elastic.co/u/Russell_Fulton)\
**Post date:** [April 5, 2021, 10:59pm UTC](https://discuss.elastic.co/t/doing-periodic-searches-via-api/269268/3 "2021-04-05T22:59:37Z")

</div>

Thanks David!

I have timestamps (to second resolution) but timestamps may coincide. That is why I am looking at alternatives. The data is coming off a network Intrusion detection system and we regularly see rates of 100 events per second or more.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 6, 2021, 2:57am UTC](https://discuss.elastic.co/t/doing-periodic-searches-via-api/269268/4 "2021-04-06T02:57:10Z")

</div>

I don't see why this would not work. You said that you need to pick new entries, right?

Here is the pseudo code of what I'd do.

```
start = 0

For
  current = new Date()
  Range from start to current.
  Start = current 

```

Note that `to` parameter is `lt` and `from` is `gte`.

Why this would not work?

---

<div class="post-metadata">

**Author:** ![Russell\_Fulton](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/russell_fulton/32/62888_2.png) [@Russell\_Fulton](https://discuss.elastic.co/u/Russell_Fulton)\
**Post date:** [April 6, 2021, 3:28am UTC](https://discuss.elastic.co/t/doing-periodic-searches-via-api/269268/5 "2021-04-06T03:28:48Z")

</div>

yes, that works if you set the end of the range with `lt` so you don't include the current second which will be incomplete.

I just think it is cleaner and more understandable to use a strictly increasing field if you have one.

What I was checking was "Is there any downsides to using \_id".

I may well use @timestamp.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 6, 2021, 3:51am UTC](https://discuss.elastic.co/t/doing-periodic-searches-via-api/269268/6 "2021-04-06T03:51:03Z")

</div>

I don't think you can use `_id` as it's not a sequential number unless you manually generate it.

---

<div class="post-metadata">

**Author:** ![Russell\_Fulton](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/russell_fulton/32/62888_2.png) [@Russell\_Fulton](https://discuss.elastic.co/u/Russell_Fulton)\
**Post date:** [April 6, 2021, 4:04am UTC](https://discuss.elastic.co/t/doing-periodic-searches-via-api/269268/7 "2021-04-06T04:04:31Z")

</div>

Thanks David! In that case I will go with @timestamp. (which is what I am using for testing right this minute : )

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 4, 2021, 4:05am UTC](https://discuss.elastic.co/t/doing-periodic-searches-via-api/269268/8 "2021-05-04T04:05:05Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.