I'm attempting read all data from a particular index using the Scroll API(ElasticSearch.NET-7.4.2), but I end up with the following exception
# FailureReason: Unrecoverable/Unexpected BadResponse while attempting POST on http://localhost:9200/_search/scroll?pretty=true&error_trace=true
- [1] BadResponse: Node: http://localhost:9200/ Exception: OverflowException Took: 00:00:00.0490000
The code is as follows:
[ElasticsearchType]
public class SPEXEC
{
[Text(Name="event_type")]
//[JsonProperty("event_type")]
public string EventType{ get; set; }
[Text(Name="client_app_name")]
//[JsonProperty("client_app_name")]
public string ClientAppName{ get; set; }
}
var settings = new ConnectionSettings(new Uri("http://localhost:9200"))
.DefaultIndex("filebeat-7.5.2")
.EnableDebugMode()
.PrettyJson()
.RequestTimeout(TimeSpan.FromMinutes(2));
var searchResponse = client.Search<SPExec>(s => s.Index("filebeat-7.5.2-2020.03.19-000002").MatchAll().Scroll("3m").Size(1000));
while (searchResponse.Documents.Any())
{
var resp = new ISearchResponse<SPExec>
{
Results = searchResponse.Documents,
ElapsedMilliseconds = (int)searchResponse.Took,
};
searchResponse = client.Scroll<SPExec>("1m", searchResponse.ScrollId); // Overflow exception
}
client.ClearScroll(c => c.ScrollId(searchResponse.ScrollId));
The aim of using Scroll API is to read data from Elastic to generate reports. I don't encounter any errors when using scrolling within the Dev Console.
POST filebeat-7.5.2-2020.03.19-000002/_search?scroll=1m
{
"query": {
"match_all": {}
}
, "size": 1000
}
POST /_search/scroll
{
"scroll" : "1m",
"scroll_id" : "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAX3sWOGZwN1NKUVVUNlc1dEFxbDRSZmxtQQ=="
}
POST /_search/scroll
{
"scroll" : "1m",
"scroll_id" : "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAX3sWOGZwN1NKUVVUNlc1dEFxbDRSZmxtQQ=="
}
If there something obvious that I've missed out when using Scroll API, do point me in the right direction. I've also run the application using ElasticSearch-7.6.1 and I end up with the same exception.
I also attempted to use the search_after, which as the documentation suggests is for real-time user requests which also failed with an Overflow exception.
var searchResponse = client.Search<SPEXEC>(s => s.Index(index)
.MatchAll()
.Size(1000)
.Sort(ss => ss
.Ascending(p => p.Timestamp)
.Descending(p => p.ElasticID)));
while (true)
{
if (searchResponse.Documents.Any())
{
hitTotal += searchResponse.Hits.Count;
var searchAfter = searchResponse.Hits.Last().Sorts.ToArray();
searchResponse = client.Search<SPEXEC>(s => s.Index(index)
.MatchAll()
.Size(1000)
.Sort(ss => ss
.Ascending(p => p.Timestamp)
.Descending(p => p.ElasticID))
.SearchAfter(searchAfter));
}
else
{
System.Threading.Thread.Sleep(5 * 60 * 1000);
}
}