Doing periodic searches via API

I have an application where I need to run searches on an index periodically picking up "new" entries from the index and I am wondering the best way of doing this.

I assume that _id is serial and simply saving the _id of the last document in each run and doing the next search for _id greater than my saved value should work. Correct ?

Is there a better way to do this?

What about adding a timestamp to each document and select documents added after a given date?

Thanks David!

I have timestamps (to second resolution) but timestamps may coincide. That is why I am looking at alternatives. The data is coming off a network Intrusion detection system and we regularly see rates of 100 events per second or more.

I don't see why this would not work. You said that you need to pick new entries, right?

Here is the pseudo code of what I'd do.

start = 0

  current = new Date()
  Range from start to current.
  Start = current 

Note that to parameter is lt and from is gte.

Why this would not work?

yes, that works if you set the end of the range with lt so you don't include the current second which will be incomplete.

I just think it is cleaner and more understandable to use a strictly increasing field if you have one.

What I was checking was "Is there any downsides to using _id".

I may well use @timestamp.

I don't think you can use _id as it's not a sequential number unless you manually generate it.

Thanks David! In that case I will go with @timestamp. (which is what I am using for testing right this minute : )

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.