Hi There, hope you all well.
Our setup: our program on dotnet full framework using NEST 7.9 and elastic search 7.9. Changed elasticsearch.yml http.max_content_length: 1000mb and using plugin ingest attachment for parsing files. As well as set the attachment.content be TermVector with position offsets
Our project is using elastic search to index physical files like .pdf, docx, .csv ... and some of them can be large. We are now at a stage to stress test the system and found that when index more than 350MB Elasticsearch.Net throws 'System.OutOfMemoryException' , the exception is pasted at the bottom.
Just want to know if we have hit any limitations of elastic search in term of size that we can index in one go?
Is there any alternative that you can recommend to index large physical files?
Thanks!
Kiet Tran
Error: -----------------
FailureReason: Unrecoverable/Unexpected BadRequest while attempting PUT on http://localhost:9200/ktr/_doc/40554962-4e0d-4b6f-91fb-f77a7a5f4fb0?pipeline=attachmentsV2&refresh=wait_for
- [1] BadRequest: Node: http://localhost:9200/ Exception: OutOfMemoryException Took: 00:00:00.0419857
# Audit exception in step 1 BadRequest:
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.Buffers.DefaultArrayPool`1.Rent(Int32 minimumLength)
at Elasticsearch.Net.Utf8Json.Internal.BinaryUtil.EnsureCapacity(Byte[]& bytes, Int32 offset, Int32 appendLength) in C:\Users\russc\source\elasticsearch-net\src\Elasticsearch.Net\Utf8Json\Internal\BinaryUtil.cs:line 76
at Elasticsearch.Net.Utf8Json.JsonWriter.WriteString(String value) in C:\Users\russc\source\elasticsearch-net\src\Elasticsearch.Net\Utf8Json\JsonWriter.cs:line 343
at Serialize(Byte[][] , Object[] , JsonWriter& , DocumentPOCO , IJsonFormatterResolver )
at Elasticsearch.Net.Utf8Json.Resolvers.DynamicMethodAnonymousFormatter`1.Serialize(JsonWriter& writer, T value, IJsonFormatterResolver formatterResolver) in C:\Users\russc\source\elasticsearch-net\src\Elasticsearch.Net\Utf8Json\Resolvers\DynamicObjectResolver.cs:line 1708
at Elasticsearch.Net.Utf8Json.JsonSerializer.SerializeUnsafe[T](T value, IJsonFormatterResolver resolver) in C:\Users\russc\source\elasticsearch-net\src\Elasticsearch.Net\Utf8Json\JsonSerializer.cs:line 167
at Elasticsearch.Net.Utf8Json.JsonSerializer.Serialize[T](Stream stream, T value, IJsonFormatterResolver resolver) in C:\Users\russc\source\elasticsearch-net\src\Elasticsearch.Net\Utf8Json\JsonSerializer.cs:line 113
at Nest.DefaultHighLevelSerializer.Serialize[T](T data, Stream writableStream, SerializationFormatting formatting) in C:\Users\russc\source\elasticsearch-net\src\Nest\CommonAbstractions\SerializationBehavior\DefaultHighLevelSerializer.cs:line 40
at Elasticsearch.Net.DiagnosticsSerializerProxy.Serialize[T](T data, Stream stream, SerializationFormatting formatting) in C:\Users\russc\source\elasticsearch-net\src\Elasticsearch.Net\Serialization\DiagnosticsSerializerProxy.cs:line 100
at Nest.IndexDescriptor`1.Nest.IProxyRequest.WriteJson(IElasticsearchSerializer sourceSerializer, Stream stream, SerializationFormatting formatting) in C:\Users\russc\source\elasticsearch-net\src\Nest\Document\Single\Index\IndexRequest.cs:line 39
at Nest.ProxyRequestFormatterBase`2.Serialize(JsonWriter& writer, TRequestInterface value, IJsonFormatterResolver formatterResolver) in C:\Users\russc\source\elasticsearch-net\src\Nest\CommonAbstractions\SerializationBehavior\JsonFormatters\ProxyRequestFormatterBase.cs:line 43
at Elasticsearch.Net.Utf8Json.JsonSerializer.SerializeUnsafe[T](T value, IJsonFormatterResolver resolver) in C:\Users\russc\source\elasticsearch-net\src\Elasticsearch.Net\Utf8Json\JsonSerializer.cs:line 167
at Elasticsearch.Net.Utf8Json.JsonSerializer.Serialize[T](Stream stream, T value, IJsonFormatterResolver resolver) in C:\Users\russc\source\elasticsearch-net\src\Elasticsearch.Net\Utf8Json\JsonSerializer.cs:line 113
at Nest.DefaultHighLevelSerializer.Serialize[T](T data, Stream writableStream, SerializationFormatting formatting) in C:\Users\russc\source\elasticsearch-net\src\Nest\CommonAbstractions\SerializationBehavior\DefaultHighLevelSerializer.cs:line 40
at Elasticsearch.Net.DiagnosticsSerializerProxy.Serialize[T](T data, Stream stream, SerializationFormatting formatting) in C:\Users\russc\source\elasticsearch-net\src\Elasticsearch.Net\Serialization\DiagnosticsSerializerProxy.cs:line 100
at Elasticsearch.Net.SerializableData`1.Write(Stream writableStream, IConnectionConfigurationValues settings) in C:\Users\russc\source\elasticsearch-net\src\Elasticsearch.Net\Transport\SerializableData.cs:line 31
at Elasticsearch.Net.HttpWebRequestConnection.Request[TResponse](RequestData requestData) in C:\Users\russc\source\elasticsearch-net\src\Elasticsearch.Net\Connection\HttpWebRequestConnection.cs:line 52
at Elasticsearch.Net.RequestPipeline.CallElasticsearch[TResponse](RequestData requestData) in C:\Users\russc\source\elasticsearch-net\src\Elasticsearch.Net\Transport\Pipeline\RequestPipeline.cs:line 172
at Elasticsearch.Net.Transport`1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters) in C:\Users\russc\source\elasticsearch-net\src\Elasticsearch.Net\Transport\Transport.cs:line 77