Limit in fetching documents through rest API

[Gauti (Gautham)]

Hi All,

I'm trying to fetch documents from elasticsearch using rest api but it looks like i can pull down only limited set of documents, below is the query i'm using and i'm getting very limited documents eventhough my index has more.

http://itdashboard:9200/inc-ticket/_search

I have tried the same in dev tools as well

GET /inc_ticket/_search

Is there any documents limit restriction applied by default?

Please advice.

Thanks
Gautham

[dadoonet (David Pilato)]

You can use:

• the size and from parameters to display by default up to 10000 records to your users. If you want to change this limit, you can change index.max_result_window setting but be aware of the consequences (ie memory).
• the search after feature to do deep pagination.
• the Scroll API if you want to extract a resultset to be consumed by another tool later.

[Gauti (Gautham)]

My main objective is to retrieve data for last 30 days, is this something achievable?

like the size
http://itdashboard:9200/inc-ticket/_search?size=10000

Can i add range
http://itdashboard:9200/inc-ticket/_search?range=now-30d

Is the above range type search is possible?

Thanks
Gautham

[dadoonet (David Pilato)]

There are 2 questions here:

1. How I can select the last 30 days of records?
2. How I can retrieve all the documents matching this selection?

@Gauti

My main objective is to retrieve data for last 30 days, is this something achievable?

The answer is YES but:

If the number of records to retrieve is higher than 10000, you will need to check the links I shared in my previous answer.
If under 10000, you can use a range query with something like:

GET /_search/size=10000&q=date%3A%5Bnow-30d%2Fd%20TO%20now%2Fd%5D

This is the URL encoded version of this:

GET /_search/size=10000&q=date:[now-30d/d TO now/d]

But I always prefer using the QueryDSL:

GET /_search
{
  "size": 10000,
  "query": {
    "range": {
      "date": {
        "gte": "now-1d/d",
        "lte": "now/d"
      }
    }
  }
}