Cluster extremely slow after many bulk indexes

Elastic Stack Elasticsearch
1

Hi,

I have about 1500 processes, each issuing a bulk index request of 500
documents. These 1500 processes run in parallel (well, in batches of 600
or so), and they complete in just a couple of minutes. In essence, the Es
cluster is being hit with 1500 bulk index requests, each with 500
documents, over the course of a few minutes.

First issue is that not all the bulk index requests return a proper
response. This may be a client and/or proxy issue. I opened a ticket with
Tire: https://github.com/karmi/tire/issues/740

The second problem is that after issuing all these bulk requests, the Es
cluster becomes very slow respond to requests. For instance, doing a
search simply to query the total number of documents in the index, takes up
to 18 seconds.

My setup is a 2 node cluster: 30 shards, 0 replicas. Each node has 8
processors and 7.5 GB of memory. While the bulk indexing is going on, the
machines' cpu doesn't go above 60% utilization. The load doesn't go above
3. In other words, the machines look like they aren't even breaking a
sweat.

Why does the cluster become unresponsive (err, very slow) then?

Even after the cpu utilization drops significantly (http://d.pr/i/PZGg), a
simple search (curl -X GET
'http://localhost:9200/unt_production/_search?pretty') still takes 17
seconds to complete.

Thanks for the help.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Hi Christopher,

This situation might be caused by merges:

They're asynchronous, cause I/O load and invalidate caches.

Warmers might help the cause:

Since you seem to have bursts of indexing, I would disable the warmers
doing that, and enable them back once indexing is done. Otherwise,
warmers will slow down your indexing.

Best regards,
Radu

http://sematext.com/ -- Elasticsearch -- Solr -- Lucene

On Sun, 26 May 2013 10:17:47 -0700 (PDT)
"Christopher J. Bottaro" cjbottaro@gmail.com wrote:

Hi,

I have about 1500 processes, each issuing a bulk index request of 500
documents. These 1500 processes run in parallel (well, in batches of
600 or so), and they complete in just a couple of minutes. In
essence, the Es cluster is being hit with 1500 bulk index requests,
each with 500 documents, over the course of a few minutes.

First issue is that not all the bulk index requests return a proper
response. This may be a client and/or proxy issue. I opened a
ticket with Tire: bulk response is sometimes nil · Issue #740 · karmi/retire · GitHub

The second problem is that after issuing all these bulk requests, the
Es cluster becomes very slow respond to requests. For instance,
doing a search simply to query the total number of documents in the
index, takes up to 18 seconds.

My setup is a 2 node cluster: 30 shards, 0 replicas. Each node has
8 processors and 7.5 GB of memory. While the bulk indexing is going
on, the machines' cpu doesn't go above 60% utilization. The load
doesn't go above 3. In other words, the machines look like they
aren't even breaking a sweat.

Why does the cluster become unresponsive (err, very slow) then?

Even after the cpu utilization drops significantly
(http://d.pr/i/PZGg), a simple search (curl -X GET
'http://localhost:9200/unt_production/_search?pretty') still takes 17
seconds to complete.

Thanks for the help.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.