Heavy indexing cause severe delay for searching

Hajime_Takase · January 21, 2015, 2:09am

Hi,

I have a 10 indexes with 100 shards using 10 nodes which have 40 cores each.

I have a problem when indexing 1000~2000/doc or 2M~4M per sec in new index,
searching to the other old indexes happen to be very slow.

How can I prevent from indexing to interfere the searching activity?

Additional info:
/_cat/thread_pool is almost 0 in every column.
load average / cpu usage / memory usage /reading usage is low and stable.
I use bulk api of rest client not transport client.
My version is 1.4.0

thanks,

hajime

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHm3Zso%2Bu0H_uG3Dw33Ubk7OxJYU8CLdypmASPZJiwto5CM_0w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

warkolm · January 21, 2015, 3:12am

Is that 100 shards per index? If to it's too many and you should drop that
to 10-20 per index.

Check your hot threads, it might show something.

On 21 January 2015 at 15:09, Hajime placeofnomemories@gmail.com wrote:

Hi,

I have a 10 indexes with 100 shards using 10 nodes which have 40 cores
each.

I have a problem when indexing 1000~2000/doc or 2M~4M per sec in new
index, searching to the other old indexes happen to be very slow.

How can I prevent from indexing to interfere the searching activity?

Additional info:
/_cat/thread_pool is almost 0 in every column.
load average / cpu usage / memory usage /reading usage is low and stable.
I use bulk api of rest client not transport client.
My version is 1.4.0

thanks,

hajime

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHm3Zso%2Bu0H_uG3Dw33Ubk7OxJYU8CLdypmASPZJiwto5CM_0w%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHm3Zso%2Bu0H_uG3Dw33Ubk7OxJYU8CLdypmASPZJiwto5CM_0w%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-7TG5sxuCxnEjyWRWTb%2BUs9SXcREtSe-seT068L9wGmg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Hajime_Takase · January 21, 2015, 4:47am

I have 440 cores in my cluster and I find that having 100 shards are much
more faster in searching compare to 20 shards.
Does having many shards per index too costly, even I have so much cores?

On Wed, Jan 21, 2015 at 12:12 PM, Mark Walkom markwalkom@gmail.com wrote:

Is that 100 shards per index? If to it's too many and you should drop that
to 10-20 per index.

Check your hot threads, it might show something.

On 21 January 2015 at 15:09, Hajime placeofnomemories@gmail.com wrote:

Hi,

I have a 10 indexes with 100 shards using 10 nodes which have 40 cores
each.

I have a problem when indexing 1000~2000/doc or 2M~4M per sec in new
index, searching to the other old indexes happen to be very slow.

How can I prevent from indexing to interfere the searching activity?

Additional info:
/_cat/thread_pool is almost 0 in every column.
load average / cpu usage / memory usage /reading usage is low and stable.
I use bulk api of rest client not transport client.
My version is 1.4.0

thanks,

hajime

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHm3Zso%2Bu0H_uG3Dw33Ubk7OxJYU8CLdypmASPZJiwto5CM_0w%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHm3Zso%2Bu0H_uG3Dw33Ubk7OxJYU8CLdypmASPZJiwto5CM_0w%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-7TG5sxuCxnEjyWRWTb%2BUs9SXcREtSe-seT068L9wGmg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-7TG5sxuCxnEjyWRWTb%2BUs9SXcREtSe-seT068L9wGmg%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHm3ZsopXSg6k_-iygZMRSrkx5x5c-u8bbLtrk_uxYp24Og%2B-A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

dadoonet · January 21, 2015, 5:42am

Did you try to send each bulk request on different nodes in parallel?
Or does only one node get the BULK request?

David

Le 21 janv. 2015 à 05:47, Hajime placeofnomemories@gmail.com a écrit :

I have 440 cores in my cluster and I find that having 100 shards are much more faster in searching compare to 20 shards.
Does having many shards per index too costly, even I have so much cores?

On Wed, Jan 21, 2015 at 12:12 PM, Mark Walkom markwalkom@gmail.com wrote:
Is that 100 shards per index? If to it's too many and you should drop that to 10-20 per index.

Check your hot threads, it might show something.

On 21 January 2015 at 15:09, Hajime placeofnomemories@gmail.com wrote:
Hi,

I have a 10 indexes with 100 shards using 10 nodes which have 40 cores each.

I have a problem when indexing 1000~2000/doc or 2M~4M per sec in new index, searching to the other old indexes happen to be very slow.

How can I prevent from indexing to interfere the searching activity?

Additional info:
/_cat/thread_pool is almost 0 in every column.
load average / cpu usage / memory usage /reading usage is low and stable.
I use bulk api of rest client not transport client.
My version is 1.4.0

thanks,

hajime

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHm3Zso%2Bu0H_uG3Dw33Ubk7OxJYU8CLdypmASPZJiwto5CM_0w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-7TG5sxuCxnEjyWRWTb%2BUs9SXcREtSe-seT068L9wGmg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHm3ZsopXSg6k_-iygZMRSrkx5x5c-u8bbLtrk_uxYp24Og%2B-A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7499EC08-36D4-4419-A1A7-1ED63F90C5DA%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

Hajime_Takase · January 21, 2015, 6:28am

I run on parallel using REST API

On Wed, Jan 21, 2015 at 2:42 PM, David Pilato david@pilato.fr wrote:

Did you try to send each bulk request on different nodes in parallel?
Or does only one node get the BULK request?

David

Le 21 janv. 2015 à 05:47, Hajime placeofnomemories@gmail.com a écrit :

I have 440 cores in my cluster and I find that having 100 shards are much
more faster in searching compare to 20 shards.
Does having many shards per index too costly, even I have so much cores?

On Wed, Jan 21, 2015 at 12:12 PM, Mark Walkom markwalkom@gmail.com
wrote:

Is that 100 shards per index? If to it's too many and you should drop
that to 10-20 per index.

Check your hot threads, it might show something.

On 21 January 2015 at 15:09, Hajime placeofnomemories@gmail.com wrote:

Hi,

I have a 10 indexes with 100 shards using 10 nodes which have 40 cores
each.

I have a problem when indexing 1000~2000/doc or 2M~4M per sec in new
index, searching to the other old indexes happen to be very slow.

How can I prevent from indexing to interfere the searching activity?

Additional info:
/_cat/thread_pool is almost 0 in every column.
load average / cpu usage / memory usage /reading usage is low and stable.
I use bulk api of rest client not transport client.
My version is 1.4.0

thanks,

hajime

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHm3Zso%2Bu0H_uG3Dw33Ubk7OxJYU8CLdypmASPZJiwto5CM_0w%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHm3Zso%2Bu0H_uG3Dw33Ubk7OxJYU8CLdypmASPZJiwto5CM_0w%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-7TG5sxuCxnEjyWRWTb%2BUs9SXcREtSe-seT068L9wGmg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-7TG5sxuCxnEjyWRWTb%2BUs9SXcREtSe-seT068L9wGmg%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHm3ZsopXSg6k_-iygZMRSrkx5x5c-u8bbLtrk_uxYp24Og%2B-A%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHm3ZsopXSg6k_-iygZMRSrkx5x5c-u8bbLtrk_uxYp24Og%2B-A%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/7499EC08-36D4-4419-A1A7-1ED63F90C5DA%40pilato.fr
https://groups.google.com/d/msgid/elasticsearch/7499EC08-36D4-4419-A1A7-1ED63F90C5DA%40pilato.fr?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHm3ZspteZhmpf1fSo_ZcGnh_sQ%3DaoMfMqksEM8GWU674XYM-Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

dadoonet · January 21, 2015, 7:14am

But on different nodes?

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 21 janv. 2015 à 07:28, Hajime placeofnomemories@gmail.com a écrit :

I run on parallel using REST API

On Wed, Jan 21, 2015 at 2:42 PM, David Pilato david@pilato.fr wrote:
Did you try to send each bulk request on different nodes in parallel?
Or does only one node get the BULK request?

David

Le 21 janv. 2015 à 05:47, Hajime placeofnomemories@gmail.com a écrit :

I have 440 cores in my cluster and I find that having 100 shards are much more faster in searching compare to 20 shards.
Does having many shards per index too costly, even I have so much cores?

On Wed, Jan 21, 2015 at 12:12 PM, Mark Walkom markwalkom@gmail.com wrote:
Is that 100 shards per index? If to it's too many and you should drop that to 10-20 per index.

Check your hot threads, it might show something.

On 21 January 2015 at 15:09, Hajime placeofnomemories@gmail.com wrote:
Hi,

I have a 10 indexes with 100 shards using 10 nodes which have 40 cores each.

I have a problem when indexing 1000~2000/doc or 2M~4M per sec in new index, searching to the other old indexes happen to be very slow.

How can I prevent from indexing to interfere the searching activity?

Additional info:
/_cat/thread_pool is almost 0 in every column.
load average / cpu usage / memory usage /reading usage is low and stable.
I use bulk api of rest client not transport client.
My version is 1.4.0

thanks,

hajime

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHm3Zso%2Bu0H_uG3Dw33Ubk7OxJYU8CLdypmASPZJiwto5CM_0w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-7TG5sxuCxnEjyWRWTb%2BUs9SXcREtSe-seT068L9wGmg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHm3ZsopXSg6k_-iygZMRSrkx5x5c-u8bbLtrk_uxYp24Og%2B-A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7499EC08-36D4-4419-A1A7-1ED63F90C5DA%40pilato.fr.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHm3ZspteZhmpf1fSo_ZcGnh_sQ%3DaoMfMqksEM8GWU674XYM-Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/A28CFA07-BD50-433B-B87B-32379ABD0042%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

Hajime_Takase · January 21, 2015, 11:25am

Yes, it is round robin.

On Wed, Jan 21, 2015 at 4:14 PM, David Pilato david@pilato.fr wrote:

But on different nodes?

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 21 janv. 2015 à 07:28, Hajime placeofnomemories@gmail.com a écrit :

I run on parallel using REST API

On Wed, Jan 21, 2015 at 2:42 PM, David Pilato david@pilato.fr wrote:

Did you try to send each bulk request on different nodes in parallel?
Or does only one node get the BULK request?

David

Le 21 janv. 2015 à 05:47, Hajime placeofnomemories@gmail.com a écrit :

I have 440 cores in my cluster and I find that having 100 shards are much
more faster in searching compare to 20 shards.
Does having many shards per index too costly, even I have so much cores?

On Wed, Jan 21, 2015 at 12:12 PM, Mark Walkom markwalkom@gmail.com
wrote:

Is that 100 shards per index? If to it's too many and you should drop
that to 10-20 per index.

Check your hot threads, it might show something.

On 21 January 2015 at 15:09, Hajime placeofnomemories@gmail.com wrote:

Hi,

I have a 10 indexes with 100 shards using 10 nodes which have 40 cores
each.

I have a problem when indexing 1000~2000/doc or 2M~4M per sec in new
index, searching to the other old indexes happen to be very slow.

How can I prevent from indexing to interfere the searching activity?

Additional info:
/_cat/thread_pool is almost 0 in every column.
load average / cpu usage / memory usage /reading usage is low and
stable.
I use bulk api of rest client not transport client.
My version is 1.4.0

thanks,

hajime

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHm3Zso%2Bu0H_uG3Dw33Ubk7OxJYU8CLdypmASPZJiwto5CM_0w%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHm3Zso%2Bu0H_uG3Dw33Ubk7OxJYU8CLdypmASPZJiwto5CM_0w%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-7TG5sxuCxnEjyWRWTb%2BUs9SXcREtSe-seT068L9wGmg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-7TG5sxuCxnEjyWRWTb%2BUs9SXcREtSe-seT068L9wGmg%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHm3ZsopXSg6k_-iygZMRSrkx5x5c-u8bbLtrk_uxYp24Og%2B-A%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHm3ZsopXSg6k_-iygZMRSrkx5x5c-u8bbLtrk_uxYp24Og%2B-A%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/7499EC08-36D4-4419-A1A7-1ED63F90C5DA%40pilato.fr
https://groups.google.com/d/msgid/elasticsearch/7499EC08-36D4-4419-A1A7-1ED63F90C5DA%40pilato.fr?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHm3ZspteZhmpf1fSo_ZcGnh_sQ%3DaoMfMqksEM8GWU674XYM-Q%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHm3ZspteZhmpf1fSo_ZcGnh_sQ%3DaoMfMqksEM8GWU674XYM-Q%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/A28CFA07-BD50-433B-B87B-32379ABD0042%40pilato.fr
https://groups.google.com/d/msgid/elasticsearch/A28CFA07-BD50-433B-B87B-32379ABD0042%40pilato.fr?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHm3ZsrA6RVsbXvoqY7hrHm%3DQwMqRYwXs9OYo6RfMOFU9nJdzw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Hajime_Takase · January 21, 2015, 2:14pm

Using TransportClient has just solved the problem.I bulk 40000 docs per
seconds but loadaverage,searching,everything is perfect.

On Wed, Jan 21, 2015 at 8:25 PM, Hajime placeofnomemories@gmail.com wrote:

Yes, it is round robin.

On Wed, Jan 21, 2015 at 4:14 PM, David Pilato david@pilato.fr wrote:

But on different nodes?

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 21 janv. 2015 à 07:28, Hajime placeofnomemories@gmail.com a écrit :

I run on parallel using REST API

On Wed, Jan 21, 2015 at 2:42 PM, David Pilato david@pilato.fr wrote:

Did you try to send each bulk request on different nodes in parallel?
Or does only one node get the BULK request?

David

Le 21 janv. 2015 à 05:47, Hajime placeofnomemories@gmail.com a écrit :

I have 440 cores in my cluster and I find that having 100 shards are
much more faster in searching compare to 20 shards.
Does having many shards per index too costly, even I have so much cores?

On Wed, Jan 21, 2015 at 12:12 PM, Mark Walkom markwalkom@gmail.com
wrote:

Is that 100 shards per index? If to it's too many and you should drop
that to 10-20 per index.

Check your hot threads, it might show something.

On 21 January 2015 at 15:09, Hajime placeofnomemories@gmail.com
wrote:

Hi,

I have a 10 indexes with 100 shards using 10 nodes which have 40 cores
each.

I have a problem when indexing 1000~2000/doc or 2M~4M per sec in new
index, searching to the other old indexes happen to be very slow.

How can I prevent from indexing to interfere the searching activity?

Additional info:
/_cat/thread_pool is almost 0 in every column.
load average / cpu usage / memory usage /reading usage is low and
stable.
I use bulk api of rest client not transport client.
My version is 1.4.0

thanks,

hajime

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHm3Zso%2Bu0H_uG3Dw33Ubk7OxJYU8CLdypmASPZJiwto5CM_0w%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHm3Zso%2Bu0H_uG3Dw33Ubk7OxJYU8CLdypmASPZJiwto5CM_0w%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-7TG5sxuCxnEjyWRWTb%2BUs9SXcREtSe-seT068L9wGmg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-7TG5sxuCxnEjyWRWTb%2BUs9SXcREtSe-seT068L9wGmg%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHm3ZsopXSg6k_-iygZMRSrkx5x5c-u8bbLtrk_uxYp24Og%2B-A%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHm3ZsopXSg6k_-iygZMRSrkx5x5c-u8bbLtrk_uxYp24Og%2B-A%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/7499EC08-36D4-4419-A1A7-1ED63F90C5DA%40pilato.fr
https://groups.google.com/d/msgid/elasticsearch/7499EC08-36D4-4419-A1A7-1ED63F90C5DA%40pilato.fr?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHm3ZspteZhmpf1fSo_ZcGnh_sQ%3DaoMfMqksEM8GWU674XYM-Q%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHm3ZspteZhmpf1fSo_ZcGnh_sQ%3DaoMfMqksEM8GWU674XYM-Q%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/A28CFA07-BD50-433B-B87B-32379ABD0042%40pilato.fr
https://groups.google.com/d/msgid/elasticsearch/A28CFA07-BD50-433B-B87B-32379ABD0042%40pilato.fr?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHm3ZsppFBEamdR_B2LZCY3ye1yCyq_fadzo_kqqj_5yp0AmVw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

jprante · January 21, 2015, 2:43pm

100 shards per index on a 10 node cluster only "feel" like being fast,
because the file structures sizes are very small in the beginning and most
of the memory is not allocated. But they are not. You will run into trouble
much later when indexes grow and segment merges get bigger, or indexes must
be recovered, or caches are used in filters/aggregations, or when tuning
for faster search.

10 or 20 shards per index on 10 nodes are really more than sufficient in
the vast majority of cases.

Jörg

On Wed, Jan 21, 2015 at 5:47 AM, Hajime placeofnomemories@gmail.com wrote:

I have 440 cores in my cluster and I find that having 100 shards are much
more faster in searching compare to 20 shards.
Does having many shards per index too costly, even I have so much cores?

On Wed, Jan 21, 2015 at 12:12 PM, Mark Walkom markwalkom@gmail.com
wrote:

Is that 100 shards per index? If to it's too many and you should drop
that to 10-20 per index.

Check your hot threads, it might show something.

On 21 January 2015 at 15:09, Hajime placeofnomemories@gmail.com wrote:

Hi,

I have a 10 indexes with 100 shards using 10 nodes which have 40 cores
each.

I have a problem when indexing 1000~2000/doc or 2M~4M per sec in new
index, searching to the other old indexes happen to be very slow.

How can I prevent from indexing to interfere the searching activity?

Additional info:
/_cat/thread_pool is almost 0 in every column.
load average / cpu usage / memory usage /reading usage is low and stable.
I use bulk api of rest client not transport client.
My version is 1.4.0

thanks,

hajime

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHm3Zso%2Bu0H_uG3Dw33Ubk7OxJYU8CLdypmASPZJiwto5CM_0w%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHm3Zso%2Bu0H_uG3Dw33Ubk7OxJYU8CLdypmASPZJiwto5CM_0w%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-7TG5sxuCxnEjyWRWTb%2BUs9SXcREtSe-seT068L9wGmg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-7TG5sxuCxnEjyWRWTb%2BUs9SXcREtSe-seT068L9wGmg%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHm3ZsopXSg6k_-iygZMRSrkx5x5c-u8bbLtrk_uxYp24Og%2B-A%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHm3ZsopXSg6k_-iygZMRSrkx5x5c-u8bbLtrk_uxYp24Og%2B-A%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGtJcXG4vT-fbftX0S5y41yHCsDwuycmNNGSpZ0ZKPqWg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Hajime_Takase · January 22, 2015, 2:33pm

I have already built the indexes with 100 shards while data count is about
1.5 million and data volumes are about 1T(include replica).I use 10 nodes
or 400 cpu cores and everything seems to be fine for time being.I was
thinking that coordinate node might be bottle neck but it seems quite
stable and faster than fewer shards(like 20).

and most of the memory is not allocated
what do you mean by this?page cache or java heap?

segment merges get bigger
Can you give any reference for this?I might make it read only or change the
refresh interval settings.

caches are used in filters/aggregations
I don't really get this part.Why having many shards will have negative
consequences when each shards execute filter or aggregation?By the way,most
fields for the aggregations are doc_values.

On Wed, Jan 21, 2015 at 11:43 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

100 shards per index on a 10 node cluster only "feel" like being fast,
because the file structures sizes are very small in the beginning and most
of the memory is not allocated. But they are not. You will run into trouble
much later when indexes grow and segment merges get bigger, or indexes must
be recovered, or caches are used in filters/aggregations, or when tuning
for faster search.

10 or 20 shards per index on 10 nodes are really more than sufficient in
the vast majority of cases.

Jörg

On Wed, Jan 21, 2015 at 5:47 AM, Hajime placeofnomemories@gmail.com
wrote:

I have 440 cores in my cluster and I find that having 100 shards are much
more faster in searching compare to 20 shards.
Does having many shards per index too costly, even I have so much cores?

On Wed, Jan 21, 2015 at 12:12 PM, Mark Walkom markwalkom@gmail.com
wrote:

Is that 100 shards per index? If to it's too many and you should drop
that to 10-20 per index.

Check your hot threads, it might show something.

On 21 January 2015 at 15:09, Hajime placeofnomemories@gmail.com wrote:

Hi,

I have a 10 indexes with 100 shards using 10 nodes which have 40 cores
each.

I have a problem when indexing 1000~2000/doc or 2M~4M per sec in new
index, searching to the other old indexes happen to be very slow.

How can I prevent from indexing to interfere the searching activity?

Additional info:
/_cat/thread_pool is almost 0 in every column.
load average / cpu usage / memory usage /reading usage is low and
stable.
I use bulk api of rest client not transport client.
My version is 1.4.0

thanks,

hajime

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHm3Zso%2Bu0H_uG3Dw33Ubk7OxJYU8CLdypmASPZJiwto5CM_0w%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHm3Zso%2Bu0H_uG3Dw33Ubk7OxJYU8CLdypmASPZJiwto5CM_0w%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-7TG5sxuCxnEjyWRWTb%2BUs9SXcREtSe-seT068L9wGmg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-7TG5sxuCxnEjyWRWTb%2BUs9SXcREtSe-seT068L9wGmg%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHm3ZsopXSg6k_-iygZMRSrkx5x5c-u8bbLtrk_uxYp24Og%2B-A%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHm3ZsopXSg6k_-iygZMRSrkx5x5c-u8bbLtrk_uxYp24Og%2B-A%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGtJcXG4vT-fbftX0S5y41yHCsDwuycmNNGSpZ0ZKPqWg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGtJcXG4vT-fbftX0S5y41yHCsDwuycmNNGSpZ0ZKPqWg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHm3ZsqNErSxCXtNpyBq92NkbG9irW4r3Uy_jM5GOJqk07amRA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Hajime_Takase · January 23, 2015, 7:18am

I still don't get why having many shards in one index matter.Since the
index is just a logical grouping of shards or lucene threads,perhaps total
num of shards per the cluster should be more significant?For Elasticsearch
to grouping the shards cost a lot?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHm3ZsqDSevYiO7B_TE7UVV7uZRhm_oEB9%2B7R4futaufL5_BRw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

limac · January 23, 2015, 9:13am

ES will send one query request to each shard when query on this index. So, if the number of shard is too big, the number of query request will also be too big to use up all query threads.

From: elasticsearch@googlegroups.com [mailto:elasticsearch@googlegroups.com] On Behalf Of Hajime
Sent: Friday, January 23, 2015 3:19 PM
To: elasticsearch@googlegroups.com
Subject: Re: Heavy indexing cause severe delay for searching

I still don't get why having many shards in one index matter.Since the index is just a logical grouping of shards or lucene threads,perhaps total num of shards per the cluster should be more significant?For Elasticsearch to grouping the shards cost a lot?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com .
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHm3ZsqDSevYiO7B_TE7UVV7uZRhm_oEB9%2B7R4futaufL5_BRw%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAHm3ZsqDSevYiO7B_TE7UVV7uZRhm_oEB9%2B7R4futaufL5_BRw%40mail.gmail.com?utm_medium=email&utm_source=footer .
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/003501d036ec%24e0868b40%24a193a1c0%24%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Topic		Replies	Views
Bulk indexing slow down when data amount increase Elasticsearch	6	2947	July 6, 2017
Elasticsearch queries are slow Elasticsearch	6	502	November 4, 2022
ElasticSearch for the huge data (around 950 million) indexing from oracle Elasticsearch	5	4100	July 6, 2017
Indexing performance Elasticsearch	6	365	July 6, 2017
Alternative bulk indexing implementations? Elasticsearch	10	2276	July 5, 2017

Heavy indexing cause severe delay for searching

hajime

hajime

Related topics