ES hangs up when sending too much index requests

Elastic Stack Elasticsearch
1

Good day,

we have faced with the ES handing by some reason, could you please take a look in it:
<nabble_img src="Untitled.png" border="0"/>

Elastic search configuration if the following:
<nabble_img src="Untitled1.png" border="0"/>

Can you comment this somehow? Do you need any additional data?

PS:

  1. Problem reproduces sporadically with different requests
  2. Log shows nothing
2

How many search requests are you sending ?

3

differently. This time only 25 requests were sent (portionally. by 5 requests per time). All requests have similar structure. userId criteria only varies.

4
  • Are the search request concurrent? If so, how many concurrent search
    requests are you sending?
  • Are you opening a new socket on each request? The OS can stall opening
    new sockets if you do that excessively over time.
  • How the machine stats on the two nodes? CPU, load?
  • How much memory do you ahve allocated on the machine, how many
    CPUs/cores? How much memory have you allocated to ES?

On Mon, Mar 26, 2012 at 9:15 PM, Ihor Hnedysh igorgnedish@gmail.com wrote:

differently. This time only 25 requests were sent (portionally. by 5
requests
per time). All requests have similar structure. userId criteria only
varies.

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/ES-hangs-up-when-sending-too-much-index-requests-tp3857628p3859097.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

5

Are the search request concurrent? If so, how many concurrent search requests are you sending?
Yes. Requests are sent in parallel. Portionally. By five requests per portion. Total 25 requests.

Are you opening a new socket on each request? The OS can stall opening new sockets if you do that excessively over time.
I'm not sure about your question since we're not using sockets directly. I will recheck it. But as far as I understand problem is that response did not came from Elasticsearch in the same time while it received it. I believe in it because request that was previously hanging is aborted right after Elasticsearch instance stopped. Also, frequency of problem reproducibility has been decreased after we have added additional data node.

How the machine stats on the two nodes? CPU, load?
No CPU load. ProcessExplorer shows some cycles performed by Elasticsearch.

How much memory do you ahve allocated on the machine, how many CPUs/cores? How much memory Have you allocated to ES?
I have standard JVM settings set. I did not tune them. PC is the following:
RAM: 6GB
CPU: 4 cores 3.2 GHz
I have win7 on board but we have current problem reproduced not only on windows machine but on Ubuntu as well.

6

up

7

Can you try and explain the problem again? How you run your tests, how many
concurrent clients, what are you running.

On Wed, Mar 28, 2012 at 11:18 AM, Ihor Hnedysh igorgnedish@gmail.comwrote:

Are the search request concurrent? If so, how many concurrent search
requests are you sending?
Yes. Requests are sent in parallel. Portionally. By five requests per
portion. Total 25 requests.

Are you opening a new socket on each request? The OS can stall opening
new
sockets if you do that excessively over time.
I'm not sure about your question since we're not using sockets directly. I
will recheck it. But as far as I understand problem is that response did
not
came from Elasticsearch in the same time while it received it. I believe
in
it because request that was previously hanging is aborted right after
Elasticsearch instance stopped. Also, frequency of problem reproducibility
has been decreased after we have added additional data node.

How the machine stats on the two nodes? CPU, load?
No CPU load. ProcessExplorer shows some cycles performed by Elasticsearch.

How much memory do you ahve allocated on the machine, how many
CPUs/cores?
How much memory Have you allocated to ES?
I have standard JVM settings set. I did not tune them. PC is the following:
RAM: 6GB
CPU: 4 cores 3.2 GHz
I have win7 on board but we have current problem reproduced not only on
windows machine but on Ubuntu as well.

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/ES-hangs-up-when-sending-too-much-index-requests-tp3857628p3863994.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

8

Ok, I will try to explain it in details.

We have about 120 tests. All the tests have some business logic and send different requests to ES. About 900 requests for all the tests.
They are run by executing a batch file. We start the batch file with tests simultaneously 2 or more times. In average, every batch file sends about 4 requests per second. Since we started 2 ones we have about 7-8 requests per second.
As a result we have about 3-7 hanged requests.
I tested it in different configurations of ES, detailed raw result are here

Note, using tests is just an easy way of reproducing of the problem. The problem exist in a real application as well.
Hope it helps.

9

We're using version 0.18.4, but the problem is reproducing on 0.19.2 as well.