Limit in fetching documents through rest API

Elastic Stack Elasticsearch
1

Hi All,

I'm trying to fetch documents from elasticsearch using rest api but it looks like i can pull down only limited set of documents, below is the query i'm using and i'm getting very limited documents eventhough my index has more.

http://itdashboard:9200/inc-ticket/_search

I have tried the same in dev tools as well

GET /inc_ticket/_search

Is there any documents limit restriction applied by default?

Please advice.

Thanks
Gautham

2

You can use:

  • the size and from parameters to display by default up to 10000 records to your users. If you want to change this limit, you can change index.max_result_window setting but be aware of the consequences (ie memory).
  • the search after feature to do deep pagination.
  • the Scroll API if you want to extract a resultset to be consumed by another tool later.
1 Like
3

My main objective is to retrieve data for last 30 days, is this something achievable?

like the size
http://itdashboard:9200/inc-ticket/_search?size=10000

Can i add range
http://itdashboard:9200/inc-ticket/_search?range=now-30d

Is the above range type search is possible?

Thanks
Gautham

4

There are 2 questions here:

  1. How I can select the last 30 days of records?
  2. How I can retrieve all the documents matching this selection?

The answer is YES but:

If the number of records to retrieve is higher than 10000, you will need to check the links I shared in my previous answer.
If under 10000, you can use a range query with something like:

GET /_search/size=10000&q=date%3A%5Bnow-30d%2Fd%20TO%20now%2Fd%5D

This is the URL encoded version of this:

GET /_search/size=10000&q=date:[now-30d/d TO now/d]

But I always prefer using the QueryDSL:

GET /_search
{
  "size": 10000,
  "query": {
    "range": {
      "date": {
        "gte": "now-1d/d",
        "lte": "now/d"
      }
    }
  }
}
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.