Point in Time API and Search After .net NEST

[NoviceESCoder]

I have been reading up on Point in time API and wanted to implement it using NEST in my .net application. However, when reading that article (.net application hyperlink), I saw the Fluent DSL example as shown below. Is there a way to find that ID without having to go to kibana on the console and doing an query search to get the id to then later place that id inside of "a-point-in-time-id"? or does "a-point-in-time-id" do that for you like is that mapped to the ID?

s => s
.PointInTime("a-point-in-time-id", p => p
.KeepAlive("1m"))

and when reading up on search_after and attempting to implement it using the search after usage for the .net client using the Fluent DSL Example. I noticed that they had they word "Project" but it does not say what "Project is in the example. What would that be exactly?

s => s
.Sort(srt => srt
    .Descending(p => p.NumberOfCommits)
    .Descending(p => p.Name)
)
.SearchAfter(
    Project.First.NumberOfCommits,
    Project.First.Name
)

Here I attempted to implement .PointInTime() and .Sort() and .SearchAfter() but got stuck.

var response = await _elasticClient.SearchAsync<EsSource>(s => s
                  .Size(3000) // must see about this
                  .Source(src => src.Includes(i => i
                                    .Fields(f => f.timestamp,
                                            fields => fields.messageTemplate,
                                            fields => fields.message)))
                  .Index("customer-simulation-es-app-logs*"
                  .Query(q => +q
                      .DateRange(dr => dr
                          .Field("@timestamp")
                              .GreaterThanOrEquals("2021-06-12T16:39:32.727-05:00")
                              .LessThanOrEquals(DateTime.Now))))
                  .PointInTime("", p => p
                                   .KeepAlive("5m"))
                  .Sort(srt => srt
                               .Ascending(p => p.timestamp))
                  .SearchAfter()

I know when you are using PIT ID you do not need the index in the search, but in the hyperlink example it does not show how you would go about implementing that. So just a bit lost on how to do so. Any pointers/guidance/tutorials would be great!

[stevejgordon (Steve Gordon)]

Hi! Thanks for this great question. This is an area where I am planning some improved documentation and some blog posts.

In the future, we will have helpers to make using PIT from NEST easier. For now, you need to manually open a PIT and close it once you are finished. You can achieve this programmatically via NEST using the OpenPointInTimeAsync and ClosePointInTimeAsync methods.

As for SearchAfter, you should provide the sort values of the last hit from the previous set of results. This indicates to Elasticsearch where you are in the result set.

Here is a short but complete example that hopefully makes this clearer:

var pitResponse = await Client.OpenPointInTimeAsync(IndexName, p => p.KeepAlive("2m"));

if (pitResponse.IsValid)
{
    var searchOne = await Client.SearchAsync<StockData>(s => s
        .Index(IndexName)
        .Size(10)
        .Query(q => q.Match(m => m.Field(f => f.Name).Query("Microsoft")))
        .PointInTime(pitResponse.Id).Sort(srt => srt.Descending(f => f.High)));

    var lastHit = searchOne.Hits.Last();

    var searchTwo = await Client.SearchAsync<StockData>(s => s
        .Index(IndexName)
        .Size(10)
        .Query(q => q.Match(m => m.Field(f => f.Name).Query("Microsoft")))
        .PointInTime(pitResponse.Id).Sort(srt => srt.Descending(f => f.High))
        .SearchAfter(lastHit.Sorts));
}

var closeResponse = await Client.ClosePointInTimeAsync(p => p.Id(pitResponse.Id));

In this example StockData is a basic type representing data in my index:

public class StockData
{
    public DateTime Date { get; init; }
    public double High { get; init; }
    public double Low { get; init; }
    public string Symbol { get; init; }
    public string Name { get; init; }
}

In the code, I first open a point in time that will be kept alive for two minutes. The response from this request includes the ID generated for the PIT. This is then passed to the first search. This search looks for Microsoft stock data, sorted with the highest values first.

Once we have the response, we can access the last hit and then in the subsequent search request we can pass the sort values from that hit into the SearchAfter method.

I hope this makes things a little clearer and answers your question?