IOException with C# client

Elastic Stack Elasticsearch
1

Hi,

I'm in the progress of creating a custom C# client for elastic search with
RestSharp http://restsharp.org/.
Everything works without a problem except searching.

I'm indexing this document:

{
"Content": "Hello!",
"User": "Arxisos"
}

And after that I start the search request:

POST http://http//localhost:9200/twitter/tweet/_search HTTP/1.1
Accept: application/json, application/xml, text/json, text/x-json,
text/javascript, text/xml, application/json
User-Agent: RestSharp 102.3.0.0
Content-Type: application/json
Host: http
Content-Length: 82
Accept-Encoding: gzip, deflate
Proxy-Connection: Keep-Alive

{
"query":{
"fuzzy":{
"User":{
"value":"Arxksos",
"min_similarity":0.1
}
}
}
}

Often that works without any problem:
{
"took":2,
"timed_out":false,
"_shards":{
"total":5,
"successful":5,
"failed":0
},
"hits":{
"total":1,
"max_score":0.3068528,
"hits":[
{
"_index":"twitter",
"_type":"tweet",
"_id":"1",
"_score":0.3068528,
"_source":{
"Content":"Hello!",
"User":"Arxisos"
}
}
]
}
}

But sometimes elasticsearch returns with an empty result:

{
"took":1,
"timed_out":false,
"_shards":{
"total":5,
"successful":5,
"failed":0
},
"hits":{
"total":0,
"max_score":null,
"hits":[]
}
}

After looking in the log file I found this:

[2011-10-31 22:07:43,368][WARN ][http.netty ] [Projector] Caught exception
while handling client http traffic, closing connection
java.io.IOException: Eine vorhandene Verbindung wurde vom Remotehost
geschlossen
at sun.nio.ch.SocketDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(Unknown Source)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)
at sun.nio.ch.IOUtil.read(Unknown Source)
at sun.nio.ch.SocketChannelImpl.read(Unknown Source)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:321)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:280)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:200)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

The exception has a german description because my operating system's
language is german.
Here is the english translation: An existing connection was closed by the
remote host.

Do you thing the error is in my client implementation? Do I make a fault
when creating the http request?
I would be happy if someone can give some tips to solve this problem.

Thanks for reading,
Steffen

2

Maybe it's important that I've done this in the config file:
gateway.type: none

3

Hey,

Hard to tell, the failure means that the client side closed the
connection. Does RestSharp opening and closing connections for each request?

-shay.banon

On Tue, Nov 1, 2011 at 12:05 AM, Arxisos arxisos@googlemail.com wrote:

Maybe it's important that I've done this in the config file:
gateway.type: none

4

I implemented the http communication myself with System.Net.WebRequest.
Internally RestSharp is using this class, too. The connection is closed
when the instance of this class is disposed.*
*
The behavior is the same even when I switch between my implementation and
RestSharp.

After experimenting a few minutes I realized that this behavior only
happens when I'm doing CRUD (put and delete) operations very frequently.
I was calling Elasticsearch with an integration test which I executed very
often, so the document was created and deleted very often per second.

Maybe my fault isn't a bug, but a expected overhead of Elasticsearch to
add the document to the index in the specific shard (I've got 5 shards and
1 replica).
While this happens the search is returning with zero elements.*
*

Do you thing this guess is plausible?

Am Dienstag, 1. November 2011 18:39:04 UTC+1 schrieb kimchy:

Hey,

Hard to tell, the failure means that the client side closed the
connection. Does RestSharp opening and closing connections for each request?

-shay.banon

On Tue, Nov 1, 2011 at 12:05 AM, Arxisos arx...@googlemail.com wrote:

Maybe it's important that I've done this in the config file:
gateway.type: none

5

Is there a chance that you end up with no documents in the index, and thats
why the result is empty? Otherwise, indexing / deleting data will not
affect search returning no results suddenly...

On Tue, Nov 1, 2011 at 10:43 PM, Arxisos arxisos@googlemail.com wrote:

I implemented the http communication myself with System.Net.WebRequest.
Internally RestSharp is using this class, too. The connection is closed
when the instance of this class is disposed.*
*
The behavior is the same even when I switch between my implementation and
RestSharp.

After experimenting a few minutes I realized that this behavior only
happens when I'm doing CRUD (put and delete) operations very frequently.
I was calling Elasticsearch with an integration test which I executed
very often, so the document was created and deleted very often per second.

Maybe my fault isn't a bug, but a expected overhead of Elasticsearch to
add the document to the index in the specific shard (I've got 5 shards and
1 replica).
While this happens the search is returning with zero elements.*
*

Do you thing this guess is plausible?

Am Dienstag, 1. November 2011 18:39:04 UTC+1 schrieb kimchy:

Hey,

Hard to tell, the failure means that the client side closed the
connection. Does RestSharp opening and closing connections for each request?

-shay.banon

On Tue, Nov 1, 2011 at 12:05 AM, Arxisos arx...@googlemail.com wrote:

Maybe it's important that I've done this in the config file:
gateway.type: none

6

No, I'm sure that there is a document in the index because I create a GET
request before starting the search to verify if a document exists.

I compared the raw request which is created by my client with a search
request created by Fiddler (HTTP Web Debugging Tool). They look the same:

POST http://localhost:9200/twitter/tweet/_search HTTP/1.1
Host: localhost:9200
Content-Length: 82

{"query": { "fuzzy": { "User": { "value": "Arxksos", "min_similarity": 0.1
} } } }

But the response is different sometimes.

That's the order of operations which I'm executing in the integration test:

  • PUT /twitter/tweet/1
  • GET /twitter/tweet/1 (verify if document exists)
  • POST /twitter/tweet/_search
  • POST /twitter/tweet/_count (verify again if document exists - but count
    is always 1 although search returns zero)
  • DELETE /twitter/tweet/1

I really don't know how this failure is produced. But what I know is that
all works without any problem when I don't make the DELETE request at the
end.

7

This probably has to do with refreshing aspect. When you index a document,
its not visible immediately for search (that the near real time aspect of
it). By default, the index will be "refreshed" every second to make the
changes happened during that second visible for search. You can force a
refresh (at least for your test) by using the refresh API:
Elasticsearch Platform — Find real-time answers at scale | Elastic.

On Wed, Nov 2, 2011 at 10:31 AM, Arxisos arxisos@googlemail.com wrote:

No, I'm sure that there is a document in the index because I create a GET
request before starting the search to verify if a document exists.

I compared the raw request which is created by my client with a search
request created by Fiddler (HTTP Web Debugging Tool). They look the same:

POST http://localhost:9200/twitter/tweet/_search HTTP/1.1
Host: localhost:9200
Content-Length: 82

{"query": { "fuzzy": { "User": { "value": "Arxksos", "min_similarity": 0.1
} } } }

But the response is different sometimes.

That's the order of operations which I'm executing in the integration test:

  • PUT /twitter/tweet/1
  • GET /twitter/tweet/1 (verify if document exists)
  • POST /twitter/tweet/_search
  • POST /twitter/tweet/_count (verify again if document exists - but count
    is always 1 although search returns zero)
  • DELETE /twitter/tweet/1

I really don't know how this failure is produced. But what I know is that
all works without any problem when I don't make the DELETE request at the
end.

8

Wow, calling refresh really solved the problem :slight_smile:
As a reward for you/community I'll publish the C# client as open source.
After that there will be a .Net client with extra-cool fluent syntax and
convention over configuration paradigm.

*Offtopic: Maybe you'll find the syntax sooo cool that you want to create a
similar java client :smiley: *

Thanks for helping!