ElasticSearch .NET Slow System.IO.MemoryStream:set_Capacity

Elastic Stack Elasticsearch
1

Hi,

Have anyone using ElasticSearch 6.3 and ElasticSearch.NET and NEST libraries have experienced slow code on the following stack trace (with response time)

Nest.ElasticClient:Search: Unknown (execution time: 0 ms of 9000 ms total)
                          Nest.ElasticClient:Nest.IHighLevelToLowLevelDispatcher.Dispatch: Unknown (execution time: 0 ms of 9000 ms total)
                            Nest.LowLevelDispatch:SearchDispatch: Unknown (execution time: 0 ms of 9000 ms total)
                              Elasticsearch.Net.Transport`1:Request: Unknown (execution time: 0 ms of 9000 ms total)
                                Elasticsearch.Net.RequestPipeline:CallElasticsearch: Unknown (execution time: 0 ms of 9000 ms total)
                                  Elasticsearch.Net.HttpConnection:Request: Unknown (execution time: 0 ms of 9000 ms total)
                                    Elasticsearch.Net.ResponseBuilder:ToResponse: Unknown (execution time: 0 ms of 9000 ms total)
                                      Elasticsearch.Net.ResponseBuilder:SetBody: Unknown (execution time: 0 ms of 9000 ms total)
                                        System.IO.Stream:InternalCopyTo: Unknown (execution time: 0 ms of 8984 ms total)
                                          System.IO.MemoryStream:Write: Unknown (execution time: 0 ms of 8984 ms total)
                                            System.IO.MemoryStream:EnsureCapacity: Unknown (execution time: 0 ms of 8984 ms total)
                                              System.IO.MemoryStream:set_Capacity: Unknown (execution time: 8984 ms of 8984 ms total)

Does it have to do with the amount data returned from the search request?

Thanks,
Ajee

2

Looks to be spending a fair amount of time in ensuring sufficient capacity in a byte buffer backed MemoryStream. What version of NEST and Elasticsearch.Net are you using?

3

We used these nugets to get the libraries.
Elasticsearch.Net 6.3
NEST 6.3

4

Can you update to latest 6.x NEST and Elasticsearch.Net, which is 6.8.1 currently, and see if the problem persists? If it does, you'll need to provide more information around what you're doing in the client, and how you're capturing timings.

5

Hi forloop,

Unfortunately, upgrade is not in the options. We need to figure out what's causing this its current configuration.

We have a Application Performance Management tool that captures code execution and run time. The stack trace (with response time) was the result of a previous performance test.

I should have noted earlier that the scenario is on a web application with high concurrent users. Could the concurrent requests be having problems with run time memory allocation?

6

I mean upgrade the client libraries to 6.8.1; you don't need to update Elasticsearch too, the 6.8.1 client libraries are compatible with Elasticsearch 6.3.0. It is worth doing this to rule out a potential bug in the client that may now be fixed in a later release.

7

I guess we can give that a try. Other than that, can we do anything else to pin point the problem?

8

We would need to know much more information about exactly what queries are being sent, how the client is configured, how the APM tool is capturing code execution, etc., to be able to understand if this is a real issue in the client. There could be usage patterns such as returning very large number of results in a single response, combined with .DisableDirectStreaming(), that would require buffering in memory and large numbers of byte array allocations that could manifest in a significant amount of time in MemoryStream methods. Whether this is a real issue will however be context dependent.

First though, I'd recommend updating the client to the latest 6.x before going down this route as you may end up spending time investigating an issue that no longer exists in a later version.

9

You may have hit it right there.
We do have a scenario where there is a quite large number of results using a client connection with .DisableDirectStreaming().

From ElasticSearch Guide it says:

DisableDirectStreaming

Ensures the response bytes are always available on the ElasticsearchResponse<T>

Important

Depending on the registered serializer, this may cause the response to be buffered in memory first, potentially affecting performance.

We are using ISearchResponse but we did not change any serializer.

Let me get back to you with the test result using the latest library.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.