Elastisearch 5.5.1 with c# throw remote exception error please provide me the solution for this

[Ashish_Christian]

Elastisearch 5.5.1 will fatch data from 10000 perfactly but above 10000 it will not fatch the data using c#, and return remote exception error can you please guide me the same.

[dadoonet (David Pilato)]

Pagination is limited by default to 10k hits. And we don't recommend changing that value.

• If you want to do deep pagination you can use the search after feature: https://www.elastic.co/guide/en/elasticsearch/reference/5.6/search-request-search-after.html
• If you want to extract all results, you should look at the scroll API: https://www.elastic.co/guide/en/elasticsearch/reference/5.6/search-request-scroll.html

[Ashish_Christian]

how to get elastic search more than 10000 record from elastic server with c# query

[dadoonet (David Pilato)]

I think I answered. Not on C# though as I don't know it and its client but from an API perspective.

Did you look at scroll as I said i the dotNet client?

May be this page is relevant? https://www.elastic.co/guide/en/elasticsearch/client/net-api/5.x/sliced-scroll-search-usage.html#_object_initializer_syntax_example_18

Or may be @Martijn_Laarman can help?

[forloop (Russ Cam)]

You can use the scroll API to do this in the .NET client. The client API also exposes an IObservable<T> that can be used to scroll all the documents. Something like

var client = new ElasticClient();

var scrollAllObservable = client.ScrollAll<MyDocument>("1m", 10, sc => sc
    .Search(s => s
        .Query(q => q
            .MatchAll()
        )
    )
);

var handle = new ManualResetEvent(false);

var scrollObserver = new ScrollAllObserver<MyDocument>(
    // on next
    r =>
    {
        // do something with the documents
        r.SearchResponse.Documents
    },
    // on exception
    e =>
    {
        handle.Set();
        throw e;
    },
    // on completion
    () => handle.Set()
);

scrollAllObservable.Subscribe(scrollObserver); 
handle.WaitOne();

[Ashish_Christian]

Elastic search 5.5.1 with c# Use Scroll But Does not Search Properly How To get more than 10000 record from elastic search with c# code ?

[Ashish_Christian]

update my elastic search version but does not work proper in .net when search more than 10000 record from elastic search generate error how to solve c# with elastic search

[dadoonet (David Pilato)]

It would probably help if you share the code you wrote. Please format it using </> and check it's correctly formatted in the preview window.

[Ashish_Christian]

var brandResults = ConnectionToES.EsClient().Search<Nyu_ESProductListingFiltering>(s => s
                         .From(0)
                         .Size(354734)
                         .Index("ezmaal")
                         .Type("Nyu_ESProductListingFiltering")
                         .Query(q => q.Term(t => t.Field("categoryId").Value(categoryId)) && q.Term(t => t.Field("psmFiltering").Value(1)))
                         .Aggregations(a => a.Terms("saId", t => t.Field(p => p.SaId).Size(354734)).Terms("saName", t => t.Field(p => p.SaName).Size(354734))));

[dadoonet (David Pilato)]

Not sure what is the error but I’m pretty sure this will cause some problems

Size(354734)

[Ashish_Christian]

var brandResults = ConnectionToES.EsClient().Search<Nyu_ESProductListingFiltering>(s => s
.From(0)
.Size(354734)
.Index("ezmaal")
.Type("Nyu_ESProductListingFiltering")
.Query(q => q.Term(t => t.Field("categoryId").Value(categoryId)) && q.Term(t => t.Field("psmFiltering").Value(1)))
.Aggregations(a => a.Terms("saId", t => t.Field(p => p.SaId).Size(354734)).Terms("saName", t => t.Field(p => p.SaName).Size(354734))));

my Query is mention above , it's executes in elasticsearch1.7.2 , where it does not executes in elasticsearch5.6.3

[dadoonet (David Pilato)]

May be circuit breaker is protecting the cluster?
Try by removing size

[Ashish_Christian]

when i do not set size it's giving me only 10 record as default but i need to get all record having in elasticsearch table and it is all most 354734 + records in the table

[dadoonet (David Pilato)]

You have 2 size in your request.

One for the size of the resultset:

                     .From(0)
                     .Size(354734)

For this one, you must use scroll API as we already explained.

One for the Terms aggregation:

                     .Aggregations(a => a.Terms("saId", t => t.Field(p => p.SaId).Size(354734)).Terms("saName", t => t.Field(p => p.SaName).Size(354734))));

This one is more problematic as you may be won't have enough memory to run it.
May be it can work though.
I'd just ask myself about the use case and the way you are trying to solve it.

What are you trying to do here?

[Ashish_Christian]

Hi,

Thank you for your suggestions and code Snippet.

I have tried that on my project. but it seems to not working for me.
Below is the code i have written to perform the feature.

Please do let me know what is missing.

 var scrollAllObservable = ConnectionToES.EsClient().ScrollAll<Nyu_ESProductListingFiltering>("1m", 10, 
                sc => sc
                .Search(s => s
                .Index("ezmaal")
                .Type("Nyu_ESProductListingFiltering")
                .From(0)
                .Size(10000)
                .Query(q => q.Term(t => t.Field("categoryId").Value(categoryId)) && q.Term(t => t.Field("psmFiltering").Value(1)))
                )
            );