How to control load on ES nodes

[MJR_M]

Hi

 In our prod ES cluster, we have 5 nodes and I have to re-index all the 

documents. I am doing bulk indexing through river, and while doing this
load on ES nodes increase to 10 - 15+, and obviously this slows down the
search. Can anyone suggest what should I do in order to bulk index through
river and at the same time keep the load on nodes under control. I have to
re-index 50 million documents.

Regards
mjr

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[radu_gheorghe]

Hello,

Here are a few things I would try:

• decrease the number of bulk threads from the
  threadpoolhttp://www.elasticsearch.org/guide/reference/modules/threadpool/
• make bulk requests smaller
• if you can change the river code, you can lower the number of concurrent
  bulk requestshttps://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/action/bulk/BulkProcessor.java#L105from
  the default 1 to 0
• increase the refresh_interval
• use store throttlinghttp://www.elasticsearch.org/guide/reference/index-modules/store/,
  in case the I/O caused by merges hurts
• increase index buffer
  sizehttp://www.elasticsearch.org/guide/reference/modules/indices/-
  that should make it more gentle to I/O, in exchange for some memory

Best regards,
Radu

On Wed, Jul 10, 2013 at 2:38 AM, MJR M emjayaarr@gmail.com wrote:

Hi

 In our prod ES cluster, we have 5 nodes and I have to re-index all

the documents. I am doing bulk indexing through river, and while doing this
load on ES nodes increase to 10 - 15+, and obviously this slows down the
search. Can anyone suggest what should I do in order to bulk index through
river and at the same time keep the load on nodes under control. I have to
re-index 50 million documents.

Regards
mjr

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
http://sematext.com/ -- Elasticsearch -- Solr -- Lucene

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.