# Issues with NEST 7. "The stream with Id and Tag is disposed."

**URL:** <https://discuss.elastic.co/t/issues-with-nest-7-the-stream-with-id-and-tag-is-disposed/203690>\
**Category:** Elasticsearch\
**Created:** [October 15, 2019, 4:24pm UTC](https://discuss.elastic.co/t/issues-with-nest-7-the-stream-with-id-and-tag-is-disposed/203690 "2019-10-15T16:24:00Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![andrx](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrx/32/35028_2.png) [@andrx](https://discuss.elastic.co/u/andrx)\
**Post date:** [October 15, 2019, 4:24pm UTC](https://discuss.elastic.co/t/issues-with-nest-7-the-stream-with-id-and-tag-is-disposed/203690/1 "2019-10-15T16:24:00Z")

</div>

After update to Nest7 (basically nuget, no other changes) occasionally getting this error for a process that usually runs for 15 minutes.  
Sometimes it happens during updating documents by id, sometimes bulk operations.  
The problem disappeared when I turned off parallel processing _ParallelOptions { MaxDegreeOfParallelism = 1 }_ so, basically lowered the load.  
Have never had the issue with Nest6. _IElasticClient_ is registered as singleton via DI.  
The client talks directly to ES by IPs set in _StaticConnectionPool_

```
connectionSettings.EnableHttpCompression();
#if DEBUG
//connectionSettings.EnableDebugMode();
#endif
connectionSettings.DisableDirectStreaming();
connectionSettings.EnableHttpPipelining();
connectionSettings.PrettyJson(false);
connectionSettings.IncludeServerStackTraceOnError();

```

Any ideas as to what may have slipped during migration?  
Maybe ConnectionSettings should be adjusted?  
I'm checking whether there are multi-threading issues but as I mentioned no issues ever happened with version 6.

ES: centos7, ES 7.3.2  
client: centos7, .net core 2.0, Nest 7.3.1

Log:

```
System.AggregateException: One or more errors occurred. (The read operation failed, see inner exception.) ---> Elasticsearch.Net.UnexpectedElasticsearchClientException: The read operation failed, see inner exception. ---> System.IO.IOException: The read operation failed, see inner exception. ---> System.ObjectDisposedException: Cannot access a disposed object.

Object name: 'The stream with Id f434e91e-8fcd-4051-83fd-56ca445e5e53 and Tag is disposed.'.
at Elasticsearch.Net.RecyclableMemoryStream.CheckDisposed()
at Elasticsearch.Net.RecyclableMemoryStream.SafeRead(Byte[] buffer, Int32 offset, Int32 count, Int32& streamPosition)
at Elasticsearch.Net.RecyclableMemoryStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.MemoryStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)

---> (Inner Exception #0) Elasticsearch.Net.UnexpectedElasticsearchClientException: The read operation failed, see inner exception. ---> System.IO.IOException: The read operation failed, see inner exception. ---> System.ObjectDisposedException: Cannot access a disposed object.

--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at System.Net.Http.CurlHandler.MultiAgent.TransferDataFromRequestStream(IntPtr buffer, Int32 length, EasyRequest easy)
at System.Net.Http.CurlHandler.MultiAgent.CurlSendCallback(IntPtr buffer, UInt64 size, UInt64 nitems, IntPtr context)
--- End of inner exception stack trace ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at System.Net.Http.CurlHandler.CurlResponseStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.Stream.CopyTo(Stream destination, Int32 bufferSize)
at Elasticsearch.Net.ResponseBuilder.SetBody[TResponse](ApiCallDetails details, RequestData requestData, Stream responseStream, String mimeType)
at Elasticsearch.Net.ResponseBuilder.ToResponse[TResponse](RequestData requestData, Exception ex, Nullable`1 statusCode, IEnumerable`1 warnings, Stream responseStream, String mimeType)
at Elasticsearch.Net.HttpConnection.Request[TResponse](RequestData requestData)
at Elasticsearch.Net.RequestPipeline.CallElasticsearch[TResponse](RequestData requestData)
at Elasticsearch.Net.Transport`1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)
--- End of inner exception stack trace ---
at Elasticsearch.Net.Transport`1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)
at Elasticsearch.Net.ElasticLowLevelClient.DoRequest[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)
at Nest.ElasticClient.DoRequest[TRequest,TResponse](TRequest p, IRequestParameters parameters, Action`1 forceConfiguration)
at Nest.ElasticClient.Search[TDocument](ISearchRequest request)
at Nest.ElasticClient.Search[TDocument](Func`2 selector)
at Loader.Writers.CustomerRepository.SearchCustomer(Func`2 query)
at Loader.Loaders.EmailSentLoader.GetUpdatedCollection(DealerDto dealer, EmailSent[] batch, ConcurrentDictionary`2 bulkCollection)
at Loader.Loaders.EmailSentLoader.<>c __DisplayClass9_1.<Load>b__ 1(Int32 page)
at System.Threading.Tasks.Parallel.<>c __DisplayClass44_0`2.<PartitionerForEachWorker>b__ 1(IEnumerator& partitionState, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion)
```

---

<div class="post-metadata">

**Author:** ![andrx](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrx/32/35028_2.png) [@andrx](https://discuss.elastic.co/u/andrx)\
**Post date:** [October 15, 2019, 4:27pm UTC](https://discuss.elastic.co/t/issues-with-nest-7-the-stream-with-id-and-tag-is-disposed/203690/2 "2019-10-15T16:27:21Z")

</div>

UPDATE  
on the server side i noticed this but probably unrelated though will try again with _cluster.fault\_detection.follower\_check.timeout: "30s"_ as this didn't exist on ES6. think it was 30s as default behaviour.

```
[WARN][o.e.t.TransportService] [data-8] Received response for a request that has timed out, sent [13233ms] ago, timed out [3226ms] ago, action [internal:coordination/fault_detection/leader_check], node [{master-1}{y0ck8405SEmBxf7EMzjofQ}{G66uUm9pR0WLa5kJlLw5hA}{10.x.x.x}{10.x.x.x:9300}{m}{xpack.installed=true}], id [5834654]
```

---

<div class="post-metadata">

**Author:** ![forloop](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/forloop/32/9021_2.png) [@forloop](https://discuss.elastic.co/u/forloop)\
**Post date:** [October 16, 2019, 3:17am UTC](https://discuss.elastic.co/t/issues-with-nest-7-the-stream-with-id-and-tag-is-disposed/203690/3 "2019-10-16T03:17:42Z")

</div>

Looking at the stack trace, the internal memory stream in which the response is buffered is already disposed when attempted to be read. Can you provide a succinct example of the type of operation that you're performing wheb you get this exception?

---

<div class="post-metadata">

**Author:** ![andrx](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrx/32/35028_2.png) [@andrx](https://discuss.elastic.co/u/andrx)\
**Post date:** [October 17, 2019, 12:40am UTC](https://discuss.elastic.co/t/issues-with-nest-7-the-stream-with-id-and-tag-is-disposed/203690/4 "2019-10-17T00:40:53Z")

</div>

Hey Russ, glad to see you again

sure, I had to change the names of methods, etc.. but short/simplified looks like [this](https://gist.github.com/andrx/b87585c190916aea0e9ed177ecdb1d7e)  
let me know if you need better example. I'll try to get something even closer if this is not enough.  
_ConnectionSettings_ are mentioned above.

some numbers according to logs (mentioned in gist) during stress tests.  
If this process runs for two hours it can fetch 3,712,029 documents from ES and around 95% of them will be sent for bulk update.

two other loaders that have the same structure also get exception on Search. but there is one where it happens on Update.

I think i'll try one more test without _DisableDirectStreaming()_

---

<div class="post-metadata">

**Author:** ![andrx](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrx/32/35028_2.png) [@andrx](https://discuss.elastic.co/u/andrx)\
**Post date:** [October 18, 2019, 1:24pm UTC](https://discuss.elastic.co/t/issues-with-nest-7-the-stream-with-id-and-tag-is-disposed/203690/5 "2019-10-18T13:24:32Z")

</div>

without _DisableDirectStreaming()_ stacktrace looks a bit different:

```
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at System.Net.Http.CurlHandler.MultiAgent.TransferDataFromRequestStream(IntPtr buffer, Int32 length, EasyRequest easy)
at System.Net.Http.CurlHandler.MultiAgent.CurlSendCallback(IntPtr buffer, UInt64 size, UInt64 nitems, IntPtr context)
--- End of inner exception stack trace ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at System.Net.Http.CurlHandler.CurlResponseStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at Elasticsearch.Net.Utf8Json.JsonSerializer.FillFromStream(Stream input, Byte[]& buffer)
at Elasticsearch.Net.Utf8Json.JsonSerializer.Deserialize[T](Stream stream, IJsonFormatterResolver resolver)
at Elasticsearch.Net.DiagnosticsSerializerProxy.Deserialize[T](Stream stream)
at Elasticsearch.Net.ResponseBuilder.SetBody[TResponse](ApiCallDetails details, RequestData requestData, Stream responseStream, String mimeType)
at Elasticsearch.Net.ResponseBuilder.ToResponse[TResponse](RequestData requestData, Exception ex, Nullable`1 statusCode, IEnumerable`1 warnings, Stream responseStream, String mimeType)
at Elasticsearch.Net.HttpConnection.Request[TResponse](RequestData requestData)
at Elasticsearch.Net.RequestPipeline.CallElasticsearch[TResponse](RequestData requestData)
at Elasticsearch.Net.Transport`1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)
--- End of inner exception stack trace ---
at Elasticsearch.Net.Transport`1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)
at Loader.Writers.CustomerRepository.SearchCustomer(Func`2 query)
...

```

Both exceptions lead me [here](https://github.com/dotnet/corefx/blob/master/src/System.Net.Http/src/System/Net/Http/CurlHandler/CurlHandler.cs) and [here](https://github.com/dotnet/corefx/blob/d3911035f2ba3eb5c44310342cc1d654e42aa316/src/System.Net.Http/src/System/Net/Http/CurlHandler/CurlHandler.CurlResponseMessage.cs)

Does it mean i have issues with the box setup or i have code issues?  
If it's running slower (with just one thread) then no issues but with nest6 hadn't these problems. Does Nest6 also use _CurlResponseStream_?

That's what i see on the box:

```
# yum list installed
...
libcurl.x86_64 7.29.0-46.el7 @base
dotnet-host.x86_64 2.1.5-1 @packages-microsoft-com-prod
dotnet-hostfxr-2.0.5.x86_64	2.0.5-1 @packages-microsoft-com-prod
dotnet-runtime-2.0.5.x86_64 2.0.5-1 @packages-microsoft-com-prod
dotnet-sdk-2.1.4.x86_64 2.1.4-1 @packages-microsoft-com-prod
...
# yum list available libcurl
libcurl.x86_64 7.29.0-54.el7 @base

```

I can try to update it to .net core 2.2 and also do _yum update_ just in case though not sure if this could really help

---

<div class="post-metadata">

**Author:** ![forloop](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/forloop/32/9021_2.png) [@forloop](https://discuss.elastic.co/u/forloop)\
**Post date:** [October 19, 2019, 7:08am UTC](https://discuss.elastic.co/t/issues-with-nest-7-the-stream-with-id-and-tag-is-disposed/203690/6 "2019-10-19T07:08:13Z")

</div>

> [@andrx](#):
>
> Does it mean i have issues with the box setup or i have code issues?  
> If it's running slower (with just one thread) then no issues but with nest6 hadn't these problems. Does Nest6 also use _CurlResponseStream_ ?

The stacktrace does looks suspicious and we'll need to investigate it further. It looks to originate from reading from the `CurlResponseStream` in both cases. What's the typical size of the response? Are there any further logs/diagnostics when the exception is thrown, perhaps in OS logs?

NEST 6 uses the same `HttpConnection` implementation as 7, which will use the managed [`SocketsHttpHandler`](https://docs.microsoft.com/en-us/dotnet/api/system.net.http.socketshttphandler?view=netcore-3.0) implementation on .NET Core 2.1 and above, and the `CurlHandler` on older versions, or if managed handler feature is turned off with `AppContext.SetSwitch("System.Net.Http.UseSocketsHttpHandler", false);`.

For what it's worth, we've seen other issues related to `CurlHandler` previously, [related to socket starvation](https://github.com/elastic/elasticsearch-net/issues/2784). This looks like a separate issue here though.

---

<div class="post-metadata">

**Author:** ![andrx](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrx/32/35028_2.png) [@andrx](https://discuss.elastic.co/u/andrx)\
**Post date:** [October 21, 2019, 3:13pm UTC](https://discuss.elastic.co/t/issues-with-nest-7-the-stream-with-id-and-tag-is-disposed/203690/7 "2019-10-21T15:13:03Z")

</div>

Would you like me to open github issue rather? I opened ticket here as I thought it's just a configuration problem.

Checked the code and ec2 instance. The setting _UseSocketsHttpHandler_ is not switched off.  
Max response size (300 docs) for this loader is about 6MB.  
Unfortunately, we didn't catch any OS logs as those machines are not reused. they created only for the loader's process and then self-terminated.  
I'll try to set up something to get those.

Thank you for the link and the info about changes in .net core 2.1 on this topic!  
Meantime, I will compile the app as 2.2 as well as update the libs on AMI.  
Will let you know.

---

<div class="post-metadata">

**Author:** ![forloop](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/forloop/32/9021_2.png) [@forloop](https://discuss.elastic.co/u/forloop)\
**Post date:** [October 21, 2019, 9:39pm UTC](https://discuss.elastic.co/t/issues-with-nest-7-the-stream-with-id-and-tag-is-disposed/203690/8 "2019-10-21T21:39:39Z")

</div>

A GitHub issue may be better to track for this. Would you mind opening one?

---

<div class="post-metadata">

**Author:** ![andrx](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrx/32/35028_2.png) [@andrx](https://discuss.elastic.co/u/andrx)\
**Post date:** [October 22, 2019, 12:28am UTC](https://discuss.elastic.co/t/issues-with-nest-7-the-stream-with-id-and-tag-is-disposed/203690/9 "2019-10-22T00:28:02Z")

</div>

Sure, created: [https://github.com/elastic/elasticsearch-net/issues/4168](https://github.com/elastic/elasticsearch-net/issues/4168)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 19, 2019, 12:28am UTC](https://discuss.elastic.co/t/issues-with-nest-7-the-stream-with-id-and-tag-is-disposed/203690/10 "2019-11-19T00:28:04Z")

</div>

Closed on November 19, 2019, 12:28am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.