Retrieving all records from query search without having to use .size -ElasticsSearch

Elastic Stack Elasticsearch
1

I noticed that when running the following dsl query search on the cli console for ElasticSearch that it will only return a few of the records. If I add a "size": 3000, then I will be able to retrieve more than I originally got back from the dsl query search. Is there a way I could retrieve all the records without having to use size?

My query search:

GET /customer-simulation-es-app-logs*/_search
{
  "_source": ["@timestamp", "messageTemplate", "message"], 
  "query": {
    "range": {
      "@timestamp": {
        "gte": "2021-06-07T00:00:00Z",
        "lte": "2021-06-08T00:00:00Z"
      }
    }
    }
  }
}

If I had the keyword size I retrieve more than the first search I did above:

GET /customer-simulation-es-app-logs*/_search
{
  "_source": ["@timestamp", "messageTemplate", "message"], 
  "size": 3000, 
  "query": {
    "range": {
      "@timestamp": {
        "gte": "2021-06-07T00:00:00Z",
        "lte": "2021-06-08T00:00:00Z"
      }
    }
    }
  }
}

Here is my elasticsearch nest portion so you can get an idea of how I have it:

 var response = await _elasticClient.SearchAsync<EsSource>(s => s
                  .Size(3000) // must see about this
                  .Source(src => src.Includes(i => i
                                    .Fields(f => f.timestamp,
                                            fields => fields.messageTemplate,
                                            fields => fields.message)))
                  .Index("customer-simulation-es-app-logs*")
                  .Query(q => +q
                      .DateRange(dr => dr
                          .Field("@timestamp")
                              .GreaterThanOrEquals("2021-06-07T17:13:54.414-05:00")
                              .LessThanOrEquals(DateTime.Now))));

EsSource

 public class EsSource
    {
        [Date(Name = "@timestamp")]
        public DateTimeOffset timestamp { get; set; }
        public String messageTemplate { get; set; }
        public String message { get; set; }
    }
2

The Kibana dev tools do not support any automation for this task. You can do it manually using a Point In Time API with pagination, or you can use one of our Elasticsearch clients from a script that you write. For example, the Python client offers the scan function which will load all docs in the background.

3

I see. So with the PIT ID route using search after would the user have to keep hitting the endpoint in order to get the next batch of logs? Because I am implementing this with NEST in my .NET console application that sends those logs to the database. I have it working 100%, its just that I would like to avoid having to use .size. You mentioned about the Elasticsearch clients is another route. Does the .NET client have a function like python scan where it will load all the docs and display it?

4

@wylie I have posted my NEST portion so you can see it incase you would like to see how I have it currently set up

5

I've never used the NEST client, but here are the public docs on scrolling in NEST.

I am moving your question to the Elasticsearch forum.

6

No worries. Thanks for providing me with this link and moving me over to Elasticsearch forum :slight_smile:

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.