Doing periodic searches via API

Elastic Stack Elasticsearch
1

I have an application where I need to run searches on an index periodically picking up "new" entries from the index and I am wondering the best way of doing this.

I assume that _id is serial and simply saving the _id of the last document in each run and doing the next search for _id greater than my saved value should work. Correct ?

Is there a better way to do this?

2

What about adding a timestamp to each document and select documents added after a given date?

3

Thanks David!

I have timestamps (to second resolution) but timestamps may coincide. That is why I am looking at alternatives. The data is coming off a network Intrusion detection system and we regularly see rates of 100 events per second or more.

4

I don't see why this would not work. You said that you need to pick new entries, right?

Here is the pseudo code of what I'd do.

start = 0

For
  current = new Date()
  Range from start to current.
  Start = current

Note that to parameter is lt and from is gte.

Why this would not work?

5

yes, that works if you set the end of the range with lt so you don't include the current second which will be incomplete.

I just think it is cleaner and more understandable to use a strictly increasing field if you have one.

What I was checking was "Is there any downsides to using _id".

I may well use @timestamp.

6

I don't think you can use _id as it's not a sequential number unless you manually generate it.

7

Thanks David! In that case I will go with @timestamp. (which is what I am using for testing right this minute : )

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.