I can't RTFM on this because I can't find the documentation.
It looks like some of our queries are taking about 1 second per index shard
per index.
However, the drives are still have low utilization. Around 10% ... so I'm
trying to figure out how to improve performance. My hunch is that I can
get ES to do parallel dispatch so that it can execute more queries in
parallel which the SSDs we're using should be able to handle.
I can't RTFM on this because I can't find the documentation.
It looks like some of our queries are taking about 1 second per index
shard per index.
However, the drives are still have low utilization. Around 10% ... so I'm
trying to figure out how to improve performance. My hunch is that I can
get ES to do parallel dispatch so that it can execute more queries in
parallel which the SSDs we're using should be able to handle.
A lot of your problem depends on your setup. Can you give us an overview
of your setup, both infra and data structure?
On 30 January 2015 at 07:57, Kevin Burton <burto...@gmail.com
<javascript:>> wrote:
I can't RTFM on this because I can't find the documentation.
It looks like some of our queries are taking about 1 second per index
shard per index.
However, the drives are still have low utilization. Around 10% ... so
I'm trying to figure out how to improve performance. My hunch is that I
can get ES to do parallel dispatch so that it can execute more queries in
parallel which the SSDs we're using should be able to handle.
Each shard is queried in parallel.
But if you don't have enough threads to query multiple shards at once, then
it's not the strict definition of parallel as it has to context switch.
I can't RTFM on this because I can't find the documentation.
It looks like some of our queries are taking about 1 second per index
shard per index.
However, the drives are still have low utilization. Around 10% ... so
I'm trying to figure out how to improve performance. My hunch is that I
can get ES to do parallel dispatch so that it can execute more queries in
parallel which the SSDs we're using should be able to handle.
I assume you mean hardware threads? What I want to avoid is a
configuration setting. I want all the shards to execute in parallel. Not
totally concerned about the physical hardware mapping as in practice this
will be a few hundred nanoseconds
On Thursday, January 29, 2015 at 4:09:15 PM UTC-8, Mark Walkom wrote:
Each shard is queried in parallel.
But if you don't have enough threads to query multiple shards at once,
then it's not the strict definition of parallel as it has to context switch.
On 30 January 2015 at 11:05, Kevin Burton <burto...@gmail.com
<javascript:>> wrote:
Ha. I appreciate the feedback but this doesn't answer my question.
Does it query them sequentially or in parallel.
Using parallel dispatch can dramatically improve performance so I'm
trying to track down how this works.
and I'm aware that the documentation was there, but I couldn't find
anything where it explains how the client executes requests.
I might just have to dive into the source or get on the dev list.
On Thursday, January 29, 2015 at 1:58:41 PM UTC-8, Mark Walkom wrote:
I can't RTFM on this because I can't find the documentation.
It looks like some of our queries are taking about 1 second per index
shard per index.
However, the drives are still have low utilization. Around 10% ... so
I'm trying to figure out how to improve performance. My hunch is that I
can get ES to do parallel dispatch so that it can execute more queries in
parallel which the SSDs we're using should be able to handle.
I assume you mean hardware threads? What I want to avoid is a
configuration setting. I want all the shards to execute in parallel. Not
totally concerned about the physical hardware mapping as in practice this
will be a few hundred nanoseconds
On Thursday, January 29, 2015 at 4:09:15 PM UTC-8, Mark Walkom wrote:
Each shard is queried in parallel.
But if you don't have enough threads to query multiple shards at once,
then it's not the strict definition of parallel as it has to context switch.
I can't RTFM on this because I can't find the documentation.
It looks like some of our queries are taking about 1 second per index
shard per index.
However, the drives are still have low utilization. Around 10% ... so
I'm trying to figure out how to improve performance. My hunch is that I
can get ES to do parallel dispatch so that it can execute more queries in
parallel which the SSDs we're using should be able to handle.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
