How does ES do parallel dispatch for searches?

I can't RTFM on this because I can't find the documentation.

It looks like some of our queries are taking about 1 second per index shard
per index.

However, the drives are still have low utilization. Around 10% ... so I'm
trying to figure out how to improve performance. My hunch is that I can
get ES to do parallel dispatch so that it can execute more queries in
parallel which the SSDs we're using should be able to handle.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d7088d0d-15fa-4925-a16e-fd7152bfccf1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

ES queries each applicable shard when processing a query. The docs are here

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index.html

A lot of your problem depends on your setup. Can you give us an overview of
your setup, both infra and data structure?

On 30 January 2015 at 07:57, Kevin Burton burtonator@gmail.com wrote:

I can't RTFM on this because I can't find the documentation.

It looks like some of our queries are taking about 1 second per index
shard per index.

However, the drives are still have low utilization. Around 10% ... so I'm
trying to figure out how to improve performance. My hunch is that I can
get ES to do parallel dispatch so that it can execute more queries in
parallel which the SSDs we're using should be able to handle.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/d7088d0d-15fa-4925-a16e-fd7152bfccf1%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/d7088d0d-15fa-4925-a16e-fd7152bfccf1%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8%3DTdRrtj3BC6AyaTybqyzMvHF%3DBr4kgES9%3Dt%2Bin0hi%2Bg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Ha. I appreciate the feedback but this doesn't answer my question.

Does it query them sequentially or in parallel.

Using parallel dispatch can dramatically improve performance so I'm trying
to track down how this works.

and I'm aware that the documentation was there, but I couldn't find
anything where it explains how the client executes requests.

I might just have to dive into the source or get on the dev list.

On Thursday, January 29, 2015 at 1:58:41 PM UTC-8, Mark Walkom wrote:

ES queries each applicable shard when processing a query. The docs are
here -
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index.html

A lot of your problem depends on your setup. Can you give us an overview
of your setup, both infra and data structure?

On 30 January 2015 at 07:57, Kevin Burton <burto...@gmail.com
<javascript:>> wrote:

I can't RTFM on this because I can't find the documentation.

It looks like some of our queries are taking about 1 second per index
shard per index.

However, the drives are still have low utilization. Around 10% ... so
I'm trying to figure out how to improve performance. My hunch is that I
can get ES to do parallel dispatch so that it can execute more queries in
parallel which the SSDs we're using should be able to handle.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/d7088d0d-15fa-4925-a16e-fd7152bfccf1%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/d7088d0d-15fa-4925-a16e-fd7152bfccf1%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/716f62f0-afe5-4221-a252-b57ad7bf296a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Each shard is queried in parallel.
But if you don't have enough threads to query multiple shards at once, then
it's not the strict definition of parallel as it has to context switch.

On 30 January 2015 at 11:05, Kevin Burton burtonator@gmail.com wrote:

Ha. I appreciate the feedback but this doesn't answer my question.

Does it query them sequentially or in parallel.

Using parallel dispatch can dramatically improve performance so I'm trying
to track down how this works.

and I'm aware that the documentation was there, but I couldn't find
anything where it explains how the client executes requests.

I might just have to dive into the source or get on the dev list.

On Thursday, January 29, 2015 at 1:58:41 PM UTC-8, Mark Walkom wrote:

ES queries each applicable shard when processing a query. The docs are
here - http://www.elasticsearch.org/guide/en/elasticsearch/
reference/current/index.html

A lot of your problem depends on your setup. Can you give us an overview
of your setup, both infra and data structure?

On 30 January 2015 at 07:57, Kevin Burton burto...@gmail.com wrote:

I can't RTFM on this because I can't find the documentation.

It looks like some of our queries are taking about 1 second per index
shard per index.

However, the drives are still have low utilization. Around 10% ... so
I'm trying to figure out how to improve performance. My hunch is that I
can get ES to do parallel dispatch so that it can execute more queries in
parallel which the SSDs we're using should be able to handle.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/d7088d0d-15fa-4925-a16e-fd7152bfccf1%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/d7088d0d-15fa-4925-a16e-fd7152bfccf1%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/716f62f0-afe5-4221-a252-b57ad7bf296a%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/716f62f0-afe5-4221-a252-b57ad7bf296a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_b%2BCiyViWWzScuryad5TfjOK0NO3h3K2Sn0GYXAZmUrQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

I assume you mean hardware threads? What I want to avoid is a
configuration setting. I want all the shards to execute in parallel. Not
totally concerned about the physical hardware mapping as in practice this
will be a few hundred nanoseconds :stuck_out_tongue:

On Thursday, January 29, 2015 at 4:09:15 PM UTC-8, Mark Walkom wrote:

Each shard is queried in parallel.
But if you don't have enough threads to query multiple shards at once,
then it's not the strict definition of parallel as it has to context switch.

On 30 January 2015 at 11:05, Kevin Burton <burto...@gmail.com
<javascript:>> wrote:

Ha. I appreciate the feedback but this doesn't answer my question.

Does it query them sequentially or in parallel.

Using parallel dispatch can dramatically improve performance so I'm
trying to track down how this works.

and I'm aware that the documentation was there, but I couldn't find
anything where it explains how the client executes requests.

I might just have to dive into the source or get on the dev list.

On Thursday, January 29, 2015 at 1:58:41 PM UTC-8, Mark Walkom wrote:

ES queries each applicable shard when processing a query. The docs are
here - http://www.elasticsearch.org/guide/en/elasticsearch/
reference/current/index.html

A lot of your problem depends on your setup. Can you give us an overview
of your setup, both infra and data structure?

On 30 January 2015 at 07:57, Kevin Burton burto...@gmail.com wrote:

I can't RTFM on this because I can't find the documentation.

It looks like some of our queries are taking about 1 second per index
shard per index.

However, the drives are still have low utilization. Around 10% ... so
I'm trying to figure out how to improve performance. My hunch is that I
can get ES to do parallel dispatch so that it can execute more queries in
parallel which the SSDs we're using should be able to handle.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/d7088d0d-15fa-4925-a16e-fd7152bfccf1%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/d7088d0d-15fa-4925-a16e-fd7152bfccf1%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/716f62f0-afe5-4221-a252-b57ad7bf296a%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/716f62f0-afe5-4221-a252-b57ad7bf296a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f104c138-1b4d-4aa0-9301-e01f8e95b775%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Then each is queried in parallel.

On 30 January 2015 at 11:18, Kevin Burton burtonator@gmail.com wrote:

I assume you mean hardware threads? What I want to avoid is a
configuration setting. I want all the shards to execute in parallel. Not
totally concerned about the physical hardware mapping as in practice this
will be a few hundred nanoseconds :stuck_out_tongue:

On Thursday, January 29, 2015 at 4:09:15 PM UTC-8, Mark Walkom wrote:

Each shard is queried in parallel.
But if you don't have enough threads to query multiple shards at once,
then it's not the strict definition of parallel as it has to context switch.

On 30 January 2015 at 11:05, Kevin Burton burto...@gmail.com wrote:

Ha. I appreciate the feedback but this doesn't answer my question.

Does it query them sequentially or in parallel.

Using parallel dispatch can dramatically improve performance so I'm
trying to track down how this works.

and I'm aware that the documentation was there, but I couldn't find
anything where it explains how the client executes requests.

I might just have to dive into the source or get on the dev list.

On Thursday, January 29, 2015 at 1:58:41 PM UTC-8, Mark Walkom wrote:

ES queries each applicable shard when processing a query. The docs are
here - http://www.elasticsearch.org/guide/en/elasticsearch/referenc
e/current/index.html

A lot of your problem depends on your setup. Can you give us an
overview of your setup, both infra and data structure?

On 30 January 2015 at 07:57, Kevin Burton burto...@gmail.com wrote:

I can't RTFM on this because I can't find the documentation.

It looks like some of our queries are taking about 1 second per index
shard per index.

However, the drives are still have low utilization. Around 10% ... so
I'm trying to figure out how to improve performance. My hunch is that I
can get ES to do parallel dispatch so that it can execute more queries in
parallel which the SSDs we're using should be able to handle.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/d7088d0d-15fa-4925-a16e-fd7152bfccf1%40goo
glegroups.com
https://groups.google.com/d/msgid/elasticsearch/d7088d0d-15fa-4925-a16e-fd7152bfccf1%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/716f62f0-afe5-4221-a252-b57ad7bf296a%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/716f62f0-afe5-4221-a252-b57ad7bf296a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/f104c138-1b4d-4aa0-9301-e01f8e95b775%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/f104c138-1b4d-4aa0-9301-e01f8e95b775%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9%3DKJX16QmOnCE4jWAPNUAnGF-A70dDNDoUOhdsrqbRqQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.