Only query new documents from Elasticsearch index

Elastic Stack Elasticsearch
1

I am writing a python program which will query elasticsearch index . I am using a match all query in the while loop which looks for the new documents . First problem is that I always get all the documents every time I query however I am only interested in the new documents since the last query .

Is there a way I can only read new documents instead of running while loopAlso is there a way where elasticsearch can alert that it has received new documents since the last query . I can then query only new documents

2

There's nothing native to Elasticsearch that provides this sort of approach.

You will need to use a top hits aggregation, and track the time of the last query to use that as the field you run the agg against.

3

Most people add an @timestamp (or created_at or whatever you want to call it) field to their docs.

Then you could use a range query rather than match_all to filter the docs by when they were created. As Mark says, with Elasticsearch you have to poll, it can't push an alert to you.

1 Like
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.