Elasticsearch.Net Library for .NET Core with Docker linux container not working


(Bikash Behuria) #1

Hi ,

I am not sure if anyone reported the following issue.

  • When using ASP.NET Core with Linux Docker container , _bulk api is not working however same works fine if we run same code using Visual Studio. Below are my test code using Visual Studio 2017

Sample Code snippet for connecting ElasticSearch and Post 2 JSON using _bulk
var payload = new List();
payload.Add("{"index":{"_index":"logstash-2018.05.10","_type":"logevent"}}");
payload.Add(
"{"@timestamp":"2018-05-07T09:57:25.8993627+02:00","level":"Information","messageTemplate":"Hello, world!","message":"Hello, world!"}");

var uri = new Uri("http://localhost:9200");
var configuration = new ConnectionConfiguration(uri);
client.Bulk(PostData.MultiJson(payload));

This code works fine when using visual studio but get exception when using linux docker container . Below is the stack trace

Exception (Stack Trace)

Unsuccessful low level call on POST: /_bulk\n# Audit trail of this API call:\n - [1] BadRequest: Node: http://localhost:9200/ Took: 00:00:00.1886777\n# OriginalException: System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.Http.CurlException: Couldn't connect to server\n at System.Net.Http.CurlHandler.ThrowIfCURLEError(CURLcode error)\n at System.Net.Http.CurlHandler.MultiAgent.FinishRequest(StrongToWeakReference1 easyWrapper, CURLcode messageResult)\n --- End of inner exception stack trace ---\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\n at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult()\n at System.Net.Http.HttpClient.d__58.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\n at Elasticsearch.Net.HttpConnection.Request[TResponse](RequestData requestData)


(Russ Cam) #2

The issue looks to be related to being able to connect to Elasticsearch

Have you bound the correct Docker container ports to the host?


(Bikash Behuria) #3

Hi ,

I have now bounded with port but received JSON invalidation error . But same works fine when I run without using Docker but running in Visual studio 2017 directly. Please note we are using Serilog Elasticsearch sink , which uses Elasticsearch.Net lowlevel client. For testing puposse , I create JSON string , and supply that in Payload (List) and call bulk API with PostData.MultiJson(payload). I see the exception orgin is ElasticSearch.net.

My JSON
"{"index":{"_index":"logstash-2018.05.09","_type":"logevent"}}"
"{"@timestamp":"2018-05-09T11:49:56.7878765+02:00","level":"Information","messageTemplate":"Hello, world!","message":"Hello, world!"}"

API using bulk

DebugInformaion (when running in Docker)
2018-05-09T12:46:06.3315366Z Caught exception while preforming bulk operation to Elasticsearch: Elasticsearch.Net.Unexpe
ctedElasticsearchClientException: Invalid JSON string ---> System.Runtime.Serialization.SerializationException: Invalid
JSON string
at Elasticsearch.Net.SimpleJson.DeserializeObject(String json) in C:\Users\russ\source\elasticsearch-net-master\src\E
lasticsearch.Net\Serialization\SimpleJson.cs:line 519
at Elasticsearch.Net.SimpleJson.DeserializeObject(String json, Type type, IJsonSerializerStrategy jsonSerializerStrat
egy) in C:\Users\russ\source\elasticsearch-net-master\src\Elasticsearch.Net\Serialization\SimpleJson.cs:line 552
at Elasticsearch.Net.LowLevelRequestResponseSerializer.Deserialize(Type type, Stream stream) in C:\Users\russ\source
elasticsearch-net-master\src\Elasticsearch.Net\Serialization\LowLevelRequestResponseSerializer.cs:line 27
at Elasticsearch.Net.LowLevelRequestResponseSerializer.Deserialize[T](Stream stream) in C:\Users\russ\source\elastics
earch-net-master\src\Elasticsearch.Net\Serialization\LowLevelRequestResponseSerializer.cs:line 32
at Elasticsearch.Net.ResponseBuilder.SetSpecialTypes[TResponse](Byte[] bytes, TResponse& cs) in C:\Users\russ\source
elasticsearch-net-master\src\Elasticsearch.Net\Transport\Pipeline\ResponseBuilder.cs:line 155
at Elasticsearch.Net.ResponseBuilder.SetBody[TResponse](ApiCallDetails details, RequestData requestData, Stream respo
nseStream, String mimeType) in C:\Users\russ\source\elasticsearch-net-master\src\Elasticsearch.Net\Transport\Pipeline\Re
sponseBuilder.cs:line 93
at Elasticsearch.Net.ResponseBuilder.ToResponse[TResponse](RequestData requestData, Exception ex, Nullable1 statusCo de, IEnumerable1 warnings, Stream responseStream, String mimeType) in C:\Users\russ\source\elasticsearch-net-master\src
\Elasticsearch.Net\Transport\Pipeline\ResponseBuilder.cs:line 25
at Elasticsearch.Net.HttpConnection.Request[TResponse](RequestData requestData) in C:\Users\russ\source\elasticsearch
-net-master\src\Elasticsearch.Net\Connection\HttpConnection-CoreFx.cs:line 95
at Elasticsearch.Net.RequestPipeline.CallElasticsearch[TResponse](RequestData requestData) in C:\Users\russ\source\el
asticsearch-net-master\src\Elasticsearch.Net\Transport\Pipeline\RequestPipeline.cs:line 451
at Elasticsearch.Net.Transport1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters) in C:\Users\russ\source\elasticsearch-net-master\src\Elasticsearch.Net\Transport\Transport.cs:line 6 8 --- End of inner exception stack trace --- at Elasticsearch.Net.Transport1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters
requestParameters) in C:\Users\russ\source\elasticsearch-net-master\src\Elasticsearch.Net\Transport\Transport.cs:line 8
6
at Serilog.Sinks.Elasticsearch.ElasticsearchSink.EmitBatchChecked[T](IEnumerable1 events) in /opt/app-root/src/src/S erilog.Sinks.Elasticsearch/Sinks/ElasticSearch/ElasticSearchSink.cs:line 207 at Serilog.Sinks.Elasticsearch.ElasticsearchSink.EmitBatch(IEnumerable1 events) in /opt/app-root/src/src/Serilog.Sin
ks.Elasticsearch/Sinks/ElasticSearch/ElasticSearchSink.cs:line 75

Debuginformation when using Visual Studio (success)

Audit trail of this API call:

Request:

{"index":{"_index":"dockertest","_type":"logevent"}}
{"@timestamp":"2018-05-09T11:49:56.7878765+02:00","level":"Information","messageTemplate":"Hello, world!","message":"Hello, world!"}

Response:

{"took":6,"errors":false,"items":[{"index":{"_index":"dockertest","_type":"logevent","id":"8S7xRGMBRhr-nlOL55j","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1,"status":201}}]}
2018-05-09T12:48:16.0934649Z {"index":{"_index":"dockertest","_type":"logevent"}}
2018-05-09T12:48:16.0936111Z {"@timestamp":"2018-05-09T11:49:56.7878765+02:00","level":"Information","messageTemplate":"Hello, world!","message":"Hello, world!"}
2018-05-09T12:48:16.0944369Z ############Writing Debug Logs Ends here######
2018-05-09T12:48:16.1286254Z Successful low level call on POST: /_bulk

Audit trail of this API call:

Request:

{"index":{"_index":"dockertest","_type":"logevent"}}
{"@timestamp":"2018-05-09T11:49:56.7878765+02:00","level":"Information","messageTemplate":"Hello, world!","message":"Hello, world!"}

Response:

{"took":7,"errors":false,"items":[{"index":{"_index":"dockertest","_type":"logevent","_id":"8i7xRGMBRhr-nlOL7Zh0","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1,"status":201}}]}


(Russ Cam) #4

What version of Serilog Elasticsearch sink are you using, and what version of Elasticsearch are you running against?


(Bikash Behuria) #5

Serilog Elasticsearch Sink Version # 6.5.0

ElasticSearch.Net Version # 6.0.0

I also verified with ElasticSearch.net # 6.1.0


(Bikash Behuria) #6

Hi ,

I wanted to add one more observation.

When I am using direct REST call to _bulk endpoint , from Docker , I always getting 404 not found. But the same works fine from Windows . for GET operation like _cat/health?v , works both in Linux container and from Visual Studio.

Below are the sample code I am using for POST _bulk operation

var client = new HttpClient(new HttpClientHandler {AllowAutoRedirect = false});

    var message = new HttpRequestMessage(System.Net.Http.HttpMethod.Post,
        new Uri("http://elasticsearch-openshift-tosspoc-dev.ocp.got.volvo.net/_bulk"));
   
        
      var strString1 = "{\"index\":{\"_index\":\"bikashbulktest1\",\"_type\":\"logevent\"}} \n " +
                        "{\"@timestamp\":\"2018-05-07T09:57:25.8993627+02:00\",\"level\":\"Information\",\"messageTemplate\":\"Hello, world!\",\"message\":\"Hello, world!\"}\n";
      

  
        HttpResponseMessage responseMessage = null;
        try
        {
            message.Content = new StringContent(strString1, Encoding.UTF8, "application/json");

            responseMessage = client.SendAsync(message).Result;


        }
        catch (Exception ex)
        {
            SelfLog.WriteLine("{0}" , ex.InnerException);

        }
        SelfLog.WriteLine("{0}", responseMessage.ToString());
         return responseMessage.ToString();

(Bikash Behuria) #7

I found that Centos base docker image had some issue which they solved , which fixes this issue. please close this request


(system) #8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.