Unrecoverable/unexpected badrequest while attempting post

Elastic Stack Elasticsearch
1

I have created a sample application using NEST to index files. It simply reads files from the specified location and does bulk indexing. It converts the files into base64 string (this is our requirement) before indexing.

After indexing few files I'm getting Out of Memory Exception

I just have one node and one shard, I'm using ES 5.1.2 and latest NEST version.

I couldnt post all information together because of limit on the post, hence I'm posting in parts

Please let me know if I'm missing anything, I can provide more information if necessary.

2

here is the exception I got

# FailureReason: Unrecoverable/Unexpected BadRequest while attempting POST on http://localhost:9200/esindex/_bulk?pipeline=attachment
 - [1] BadRequest: Node: http://localhost:9200/ Exception: OutOfMemoryException Took: 00:00:01.3659525
# Audit exception in step 1 BadRequest:
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Net.ScatterGatherBuffers.AllocateMemoryChunk(Int32 newSize)
   at System.Net.ScatterGatherBuffers.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.Net.ConnectStream.InternalWrite(Boolean async, Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)
   at System.Net.ConnectStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.IO.StreamWriter.Write(String value)
   at Newtonsoft.Json.JsonTextWriter.WriteRaw(String json)
   at Nest.BulkRequestJsonConverter.WriteJson(JsonWriter writer, Object value, JsonSerializer serializer)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeConvertable(JsonWriter writer, JsonConverter converter, Object value, JsonContract contract, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value, Type objectType)
   at Newtonsoft.Json.JsonSerializer.SerializeInternal(JsonWriter jsonWriter, Object value, Type objectType)
   at Nest.JsonNetSerializer.Serialize(Object data, Stream writableStream, SerializationFormatting formatting)
   at Elasticsearch.Net.PostData`1.Write(Stream writableStream, IConnectionConfigurationValues settings)
   at Elasticsearch.Net.HttpConnection.Request[TReturn](RequestData requestData)
   at Elasticsearch.Net.RequestPipeline.CallElasticsearch[TReturn](RequestData requestData)
   at Elasticsearch.Net.Transport`1.Request[TReturn](HttpMethod method, String path, PostData`1 data, IRequestParameters requestParameters)
# Inner Exception: Exception of type 'System.OutOfMemoryException' was thrown.
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Net.ScatterGatherBuffers.AllocateMemoryChunk(Int32 newSize)
   at System.Net.ScatterGatherBuffers.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.Net.ConnectStream.InternalWrite(Boolean async, Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)
   at System.Net.ConnectStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.IO.StreamWriter.Write(String value)
   at Newtonsoft.Json.JsonTextWriter.WriteRaw(String json)
   at Nest.BulkRequestJsonConverter.WriteJson(JsonWriter writer, Object value, JsonSerializer serializer)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeConvertable(JsonWriter writer, JsonConverter converter, Object value, JsonContract contract, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value, Type objectType)
   at Newtonsoft.Json.JsonSerializer.SerializeInternal(JsonWriter jsonWriter, Object value, Type objectType)
   at Nest.JsonNetSerializer.Serialize(Object data, Stream writableStream, SerializationFormatting formatting)
   at Elasticsearch.Net.PostData`1.Write(Stream writableStream, IConnectionConfigurationValues settings)
   at Elasticsearch.Net.HttpConnection.Request[TReturn](RequestData requestData)
   at Elasticsearch.Net.RequestPipeline.CallElasticsearch[TReturn](RequestData requestData)
   at Elasticsearch.Net.Transport`1.Request[TReturn](HttpMethod method, String path, PostData`1 data, IRequestParameters requestParameters)
3

Exception:

Elasticsearch.Net.UnexpectedElasticsearchClientException: Exception of type 'System.OutOfMemoryException' was thrown. ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Net.ScatterGatherBuffers.AllocateMemoryChunk(Int32 newSize)
   at System.Net.ScatterGatherBuffers.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.Net.ConnectStream.InternalWrite(Boolean async, Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)
   at System.Net.ConnectStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.IO.StreamWriter.Write(String value)
   at Newtonsoft.Json.JsonTextWriter.WriteRaw(String json)
   at Nest.BulkRequestJsonConverter.WriteJson(JsonWriter writer, Object value, JsonSerializer serializer)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeConvertable(JsonWriter writer, JsonConverter converter, Object value, JsonContract contract, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value, Type objectType)
   at Newtonsoft.Json.JsonSerializer.SerializeInternal(JsonWriter jsonWriter, Object value, Type objectType)
   at Nest.JsonNetSerializer.Serialize(Object data, Stream writableStream, SerializationFormatting formatting)
   at Elasticsearch.Net.PostData`1.Write(Stream writableStream, IConnectionConfigurationValues settings)
   at Elasticsearch.Net.HttpConnection.Request[TReturn](RequestData requestData)
   at Elasticsearch.Net.RequestPipeline.CallElasticsearch[TReturn](RequestData requestData)
   at Elasticsearch.Net.Transport`1.Request[TReturn](HttpMethod method, String path, PostData`1 data, IRequestParameters requestParameters)
   --- End of inner exception stack trace ---
   at Elasticsearch.Net.Transport`1.Request[TReturn](HttpMethod method, String path, PostData`1 data, IRequestParameters requestParameters)
   at Elasticsearch.Net.ElasticLowLevelClient.DoRequest[T](HttpMethod method, String path, PostData`1 data, IRequestParameters requestParameters)
   at Elasticsearch.Net.ElasticLowLevelClient.Bulk[T](String index, PostData`1 body, Func`2 requestParameters)
   at Nest.LowLevelDispatch.BulkDispatch[T](IRequest`1 p, PostData`1 body)
   at Nest.ElasticClient.Nest.IHighLevelToLowLevelDispatcher.Dispatch[TRequest,TQueryString,TResponse](TRequest request, Func`3 responseGenerator, Func`3 dispatch)
   at Nest.ElasticClient.Nest.IHighLevelToLowLevelDispatcher.Dispatch[TRequest,TQueryString,TResponse](TRequest request, Func`3 dispatch)
   at Nest.ElasticClient.Bulk(IBulkRequest request)
4

Here is the code

var descriptor = new BulkDescriptor();
descriptor.CreateMany<FileEntry>(entries.Select(e => new FileEntry { Data = Convert.ToBase64String(File.ReadAllBytes(e)) }),
			  (bd, q) => bd.Id(idCounter++.ToString()))
			  .Index("esindex")
			  .Pipeline("attachment");
try
{
	var result = elasticClient.Bulk(descriptor);
}
catch (Exception ex)
{
	Console.WriteLine("An exception occurred: " + ex.Message.ToString());
}

I also have the following ingest pipeline added

{
  "description": "parsing base64 data",
  "processors": [
	{
		"attachment" : {
        "field" : "data"
        }
	}
  ]
}

Thanks

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.