# Improve Query Speed

**URL:** <https://discuss.elastic.co/t/improve-query-speed/4180>\
**Category:** Elasticsearch\
**Created:** [March 31, 2011, 10:08pm UTC](https://discuss.elastic.co/t/improve-query-speed/4180 "2011-03-31T22:08:52Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![Karussell1](https://avatars.discourse-cdn.com/v4/letter/k/50afbb/32.png) [@Karussell1](https://discuss.elastic.co/u/Karussell1)\
**Post date:** [March 31, 2011, 10:08pm UTC](https://discuss.elastic.co/t/improve-query-speed/4180/1 "2011-03-31T22:08:52Z")

</div>

Hi,

simple question, but I know it depends on everything.

Here are the index properties from [jetwick.com](http://jetwick.com):

- 10 mio tweets in an index with 3 shards (I know I should use one  
index per day ... I'll do that later ;))
- feeding only ~50-100 tweets per seconds (twitter search is the  
limit now), but after ~400 tweets I'm doing a hard refresh. That is  
necessary so that I can always search (for retweets, duplicates etc)  
before indexing.
- ES is started via \*\* on a 64bit jvm. The OS still has ~7GB

I would like to create warm-up-queries like it is possible in Solr.  
Can I implement this in ElasticSearch? Is there an internal event  
which tells ElasticSearch that a new lucene searcher is or should be  
used?

What other optimizations regarding jvm, cache or index settings could  
I try?

Kind Regards,  
Peter.

\*\*  
JAVA\_OPTS="-Xmx7200m -Xms7200m"  
JAVA\_OPTS="$JAVA\_OPTS -XX:+UseCompressedOops"

JAVA\_OPTS="$JAVA\_OPTS -XX:+UseParNewGC"  
JAVA\_OPTS="$JAVA\_OPTS -XX:+UseConcMarkSweepGC"  
JAVA\_OPTS="$JAVA\_OPTS -XX:+CMSParallelRemarkEnabled"  
JAVA\_OPTS="$JAVA\_OPTS -XX:SurvivorRatio=8"  
JAVA\_OPTS="$JAVA\_OPTS -XX:MaxTenuringThreshold=1"  
JAVA\_OPTS="$JAVA\_OPTS -XX:CMSInitiatingOccupancyFraction=75"  
JAVA\_OPTS="$JAVA\_OPTS -XX:+UseCMSInitiatingOccupancyOnly"  
JAVA\_OPTS="$JAVA\_OPTS -XX:+HeapDumpOnOutOfMemoryError"

---

<div class="post-metadata">

**Author:** ![K\_B](https://avatars.discourse-cdn.com/v4/letter/k/278dde/32.png) [@K\_B](https://discuss.elastic.co/u/K_B)\
**Post date:** [April 1, 2011, 2:22pm UTC](https://discuss.elastic.co/t/improve-query-speed/4180/2 "2011-04-01T14:22:20Z")

</div>

Hi,

I've seen increase in performance using a index-refresh request when  
many new things were entered, e.g.:

client.admin().indices().refresh(new RefreshRequest(workingIndex));

(is this the hard refresh you were mentioning?)

I also noticed that ES caches queries, so in case you need certein  
things very often you could just send the queries and then they should  
be cached from that on;

On 1 Apr., 00:08, Karussell [tableyourt...@googlemail.com](mailto:tableyourt...@googlemail.com) wrote:

> Hi,
> 
> simple question, but I know it depends on everything.
> 
> Here are the index properties from [jetwick.com](http://jetwick.com):
> 
> - 10 mio tweets in an index with 3 shards (I know I should use one  
> index per day ... I'll do that later ;))
> - feeding only ~50-100 tweets per seconds (twitter search is the  
> limit now), but after ~400 tweets I'm doing a hard refresh. That is  
> necessary so that I can always search (for retweets, duplicates etc)  
> before indexing.
> - ES is started via \*\* on a 64bit jvm. The OS still has ~7GB
> 
> I would like to create warm-up-queries like it is possible in Solr.  
> Can I implement this in Elasticsearch? Is there an internal event  
> which tells Elasticsearch that a new lucene searcher is or should be  
> used?
> 
> What other optimizations regarding jvm, cache or index settings could  
> I try?
> 
> Kind Regards,  
> Peter.
> 
> \*\*  
> JAVA\_OPTS="-Xmx7200m -Xms7200m"  
> JAVA\_OPTS="$JAVA\_OPTS -XX:+UseCompressedOops"
> 
> JAVA\_OPTS="$JAVA\_OPTS -XX:+UseParNewGC"  
> JAVA\_OPTS="$JAVA\_OPTS -XX:+UseConcMarkSweepGC"  
> JAVA\_OPTS="$JAVA\_OPTS -XX:+CMSParallelRemarkEnabled"  
> JAVA\_OPTS="$JAVA\_OPTS -XX:SurvivorRatio=8"  
> JAVA\_OPTS="$JAVA\_OPTS -XX:MaxTenuringThreshold=1"  
> JAVA\_OPTS="$JAVA\_OPTS -XX:CMSInitiatingOccupancyFraction=75"  
> JAVA\_OPTS="$JAVA\_OPTS -XX:+UseCMSInitiatingOccupancyOnly"  
> JAVA\_OPTS="$JAVA\_OPTS -XX:+HeapDumpOnOutOfMemoryError"

---

<div class="post-metadata">

**Author:** ![Karussell1](https://avatars.discourse-cdn.com/v4/letter/k/50afbb/32.png) [@Karussell1](https://discuss.elastic.co/u/Karussell1)\
**Post date:** [April 1, 2011, 6:12pm UTC](https://discuss.elastic.co/t/improve-query-speed/4180/3 "2011-04-01T18:12:17Z")

</div>

> I've seen increase in performance using a index-refresh request when  
> many new things were entered, e.g.:
> 
> client.admin().indices().refresh(new RefreshRequest(workingIndex));
> 
> (is this the hard refresh you were mentioning?)

yes, exactly. Normally you shouldn't do that cause it decreases  
indexing speed, which in my case is acceptable.

> I also noticed that ES caches queries, so in case you need certein  
> things very often you could just send the queries and then they should  
> be cached from that on

Yes, that is one option. But my problem is that timing. When should I  
send those queries?  
Every second?

I would like to make sure that the user see response times less then a  
second (although even this is a bit too much).  
I must doing something wrong. I'll investigate if the massive use of  
facets is the problematic part of the query.

BTW: I'm already using a relative high number (20s) for the  
refresh\_interval to decrease realtime enforcement.

Regards,  
Peter.

---

<div class="post-metadata">

**Author:** ![K\_B](https://avatars.discourse-cdn.com/v4/letter/k/278dde/32.png) [@K\_B](https://discuss.elastic.co/u/K_B)\
**Post date:** [April 2, 2011, 9:05am UTC](https://discuss.elastic.co/t/improve-query-speed/4180/4 "2011-04-02T09:05:22Z")

</div>

Are those requests/ searches similar somehow? You could try to see  
what the top N queries are and then cache them within the app and  
refresh this cache every N seconds/ minutes /- etc. - and instead  
return only the cached values;

In case you also use much faceting you might have a look at the  
current trunk - I saw shay did some improvements regarding facet-  
speeed there;

On 1 Apr., 20:12, Karussell [tableyourt...@googlemail.com](mailto:tableyourt...@googlemail.com) wrote:

> > I've seen increase in performance using a index-refresh request when  
> > many new things were entered, e.g.:
> 
> > client.admin().indices().refresh(new RefreshRequest(workingIndex));
> 
> > (is this the hard refresh you were mentioning?)
> 
> yes, exactly. Normally you shouldn't do that cause it decreases  
> indexing speed, which in my case is acceptable.
> 
> > I also noticed that ES caches queries, so in case you need certein  
> > things very often you could just send the queries and then they should  
> > be cached from that on
> 
> Yes, that is one option. But my problem is that timing. When should I  
> send those queries?  
> Every second?
> 
> I would like to make sure that the user see response times less then a  
> second (although even this is a bit too much).  
> I must doing something wrong. I'll investigate if the massive use of  
> facets is the problematic part of the query.
> 
> BTW: I'm already using a relative high number (20s) for the  
> refresh\_interval to decrease realtime enforcement.
> 
> Regards,  
> Peter.

---

<div class="post-metadata">

**Author:** ![K\_B](https://avatars.discourse-cdn.com/v4/letter/k/278dde/32.png) [@K\_B](https://discuss.elastic.co/u/K_B)\
**Post date:** [April 2, 2011, 9:11am UTC](https://discuss.elastic.co/t/improve-query-speed/4180/5 "2011-04-02T09:11:50Z")

</div>

PS: also keep in mind that ES is still lucene under the hood;

So I think these tips might work there, too:

[http://wiki.apache.org/lucene-java/ImproveSearchingSpeed](http://wiki.apache.org/lucene-java/ImproveSearchingSpeed)

I mean OS-Tuning (swappiness can occur even if fixed java memory size)  
and more usage of filters if possible;

On 2 Apr., 11:05, "K.B." [korbinian.ba...@googlemail.com](mailto:korbinian.ba...@googlemail.com) wrote:

> Are those requests/ searches similar somehow? You could try to see  
> what the top N queries are and then cache them within the app and  
> refresh this cache every N seconds/ minutes /- etc. - and instead  
> return only the cached values;
> 
> In case you also use much faceting you might have a look at the  
> current trunk - I saw shay did some improvements regarding facet-  
> speeed there;
> 
> On 1 Apr., 20:12, Karussell [tableyourt...@googlemail.com](mailto:tableyourt...@googlemail.com) wrote:
> 
> > > I've seen increase in performance using a index-refresh request when  
> > > many new things were entered, e.g.:
> 
> > > client.admin().indices().refresh(new RefreshRequest(workingIndex));
> 
> > > (is this the hard refresh you were mentioning?)
> 
> > yes, exactly. Normally you shouldn't do that cause it decreases  
> > indexing speed, which in my case is acceptable.
> 
> > > I also noticed that ES caches queries, so in case you need certein  
> > > things very often you could just send the queries and then they should  
> > > be cached from that on
> 
> > Yes, that is one option. But my problem is that timing. When should I  
> > send those queries?  
> > Every second?
> 
> > I would like to make sure that the user see response times less then a  
> > second (although even this is a bit too much).  
> > I must doing something wrong. I'll investigate if the massive use of  
> > facets is the problematic part of the query.
> 
> > BTW: I'm already using a relative high number (20s) for the  
> > refresh\_interval to decrease realtime enforcement.
> 
> > Regards,  
> > Peter.

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [April 3, 2011, 5:56am UTC](https://discuss.elastic.co/t/improve-query-speed/4180/6 "2011-04-03T05:56:22Z")

</div>

There isn't an option to have warm up queries when a new reader is created in ES currently. This one is tricky to get right (you want to do it only when you really want to), but its certainly planned.  
On Friday, April 1, 2011 at 12:08 AM, Karussell wrote:

> Hi,
> 
> simple question, but I know it depends on everything.
> 
> Here are the index properties from [jetwick.com](http://jetwick.com):
> 
> - 10 mio tweets in an index with 3 shards (I know I should use one  
> index per day ... I'll do that later ;))
> - feeding only ~50-100 tweets per seconds (twitter search is the  
> limit now), but after ~400 tweets I'm doing a hard refresh. That is  
> necessary so that I can always search (for retweets, duplicates etc)  
> before indexing.
> - ES is started via \*\* on a 64bit jvm. The OS still has ~7GB
> 
> I would like to create warm-up-queries like it is possible in Solr.  
> Can I implement this in Elasticsearch? Is there an internal event  
> which tells Elasticsearch that a new lucene searcher is or should be  
> used?
> 
> What other optimizations regarding jvm, cache or index settings could  
> I try?
> 
> Kind Regards,  
> Peter.
> 
> \*\*  
> JAVA\_OPTS="-Xmx7200m -Xms7200m"  
> JAVA\_OPTS="$JAVA\_OPTS -XX:+UseCompressedOops"
> 
> JAVA\_OPTS="$JAVA\_OPTS -XX:+UseParNewGC"  
> JAVA\_OPTS="$JAVA\_OPTS -XX:+UseConcMarkSweepGC"  
> JAVA\_OPTS="$JAVA\_OPTS -XX:+CMSParallelRemarkEnabled"  
> JAVA\_OPTS="$JAVA\_OPTS -XX:SurvivorRatio=8"  
> JAVA\_OPTS="$JAVA\_OPTS -XX:MaxTenuringThreshold=1"  
> JAVA\_OPTS="$JAVA\_OPTS -XX:CMSInitiatingOccupancyFraction=75"  
> JAVA\_OPTS="$JAVA\_OPTS -XX:+UseCMSInitiatingOccupancyOnly"  
> JAVA\_OPTS="$JAVA\_OPTS -XX:+HeapDumpOnOutOfMemoryError"

---

<div class="post-metadata">

**Author:** ![K\_B](https://avatars.discourse-cdn.com/v4/letter/k/278dde/32.png) [@K\_B](https://discuss.elastic.co/u/K_B)\
**Post date:** [April 3, 2011, 9:18am UTC](https://discuss.elastic.co/t/improve-query-speed/4180/7 "2011-04-03T09:18:32Z")

</div>

@Shay:

does ES depend on the order of filters to apply for performance or is  
the order not important? I only know the behaviour from SOLR and there  
even the order of the filters was important;

On 3 Apr., 07:56, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:

> There isn't an option to have warm up queries when a new reader is created in ES currently. This one is tricky to get right (you want to do it only when you really want to), but its certainly planned.
> 
> On Friday, April 1, 2011 at 12:08 AM, Karussell wrote:
> 
> > Hi,
> 
> > simple question, but I know it depends on everything.
> 
> > Here are the index properties from [jetwick.com](http://jetwick.com):
> 
> > - 10 mio tweets in an index with 3 shards (I know I should use one  
> > index per day ... I'll do that later ;))
> > - feeding only ~50-100 tweets per seconds (twitter search is the  
> > limit now), but after ~400 tweets I'm doing a hard refresh. That is  
> > necessary so that I can always search (for retweets, duplicates etc)  
> > before indexing.
> > - ES is started via \*\* on a 64bit jvm. The OS still has ~7GB
> 
> > I would like to create warm-up-queries like it is possible in Solr.  
> > Can I implement this in Elasticsearch? Is there an internal event  
> > which tells Elasticsearch that a new lucene searcher is or should be  
> > used?
> 
> > What other optimizations regarding jvm, cache or index settings could  
> > I try?
> 
> > Kind Regards,  
> > Peter.
> 
> > \*\*  
> > JAVA\_OPTS="-Xmx7200m -Xms7200m"  
> > JAVA\_OPTS="$JAVA\_OPTS -XX:+UseCompressedOops"
> 
> > JAVA\_OPTS="$JAVA\_OPTS -XX:+UseParNewGC"  
> > JAVA\_OPTS="$JAVA\_OPTS -XX:+UseConcMarkSweepGC"  
> > JAVA\_OPTS="$JAVA\_OPTS -XX:+CMSParallelRemarkEnabled"  
> > JAVA\_OPTS="$JAVA\_OPTS -XX:SurvivorRatio=8"  
> > JAVA\_OPTS="$JAVA\_OPTS -XX:MaxTenuringThreshold=1"  
> > JAVA\_OPTS="$JAVA\_OPTS -XX:CMSInitiatingOccupancyFraction=75"  
> > JAVA\_OPTS="$JAVA\_OPTS -XX:+UseCMSInitiatingOccupancyOnly"  
> > JAVA\_OPTS="$JAVA\_OPTS -XX:+HeapDumpOnOutOfMemoryError"

---

<div class="post-metadata">

**Author:** ![Karussell1](https://avatars.discourse-cdn.com/v4/letter/k/50afbb/32.png) [@Karussell1](https://discuss.elastic.co/u/Karussell1)\
**Post date:** [April 4, 2011, 7:07am UTC](https://discuss.elastic.co/t/improve-query-speed/4180/8 "2011-04-04T07:07:37Z")

</div>

@K.B.: thanks for those tips, I'll have a look at the lucene level  
tuning tips again 🙂

I'm already running the ES snapshot but I'll git pulling to a more  
recent version.

> Are those requests/ searches similar somehow?

yes. they have similar facet fields, but the queries terms are totally  
different.

On 3 Apr., 07:56, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:

> There isn't an option to have warm up queries when a new reader is created in ES currently. This one is tricky to get right (you want to do it only when you really want to), but its certainly planned.

Shay, thanks for the information!

Do you know any other trick how I could improve query speed when doing  
indexing at the same time?

Or should I go with polling some default queries every 30 sec or so?

And assuming my 10 mio tweet index is split up by time - say into 3  
indices - will this improve query speed without adding hardware - not  
really right?  
(Except when I know before to which indices the query should go ...  
hmmh I'll think about that)

Regards,  
Peter

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [April 4, 2011, 5:08pm UTC](https://discuss.elastic.co/t/improve-query-speed/4180/9 "2011-04-04T17:08:32Z")

</div>

Scheduling queries every X seconds is good for now.  
On Monday, April 4, 2011 at 10:07 AM, Karussell wrote:

> @K.B.: thanks for those tips, I'll have a look at the lucene level  
> tuning tips again 🙂
> 
> I'm already running the ES snapshot but I'll git pulling to a more  
> recent version.
> 
> > Are those requests/ searches similar somehow?
> 
> yes. they have similar facet fields, but the queries terms are totally  
> different.
> 
> On 3 Apr., 07:56, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> 
> > There isn't an option to have warm up queries when a new reader is created in ES currently. This one is tricky to get right (you want to do it only when you really want to), but its certainly planned.
> 
> Shay, thanks for the information!
> 
> Do you know any other trick how I could improve query speed when doing  
> indexing at the same time?
> 
> Or should I go with polling some default queries every 30 sec or so?
> 
> And assuming my 10 mio tweet index is split up by time - say into 3  
> indices - will this improve query speed without adding hardware - not  
> really right?  
> (Except when I know before to which indices the query should go ...  
> hmmh I'll think about that)
> 
> Regards,  
> Peter

---

<div class="post-metadata">

**Author:** ![Karussell1](https://avatars.discourse-cdn.com/v4/letter/k/50afbb/32.png) [@Karussell1](https://discuss.elastic.co/u/Karussell1)\
**Post date:** [April 6, 2011, 10:25pm UTC](https://discuss.elastic.co/t/improve-query-speed/4180/10 "2011-04-06T22:25:40Z")

</div>

On 4 Apr., 19:08, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:

> Scheduling queries every X seconds is good for now.

Thanks, ok.

What else would you suggest to tune query performance? Any cache  
settings like there are in solr?

BTW: investigated facets and performance (after hacking) is now much  
better 🙂

---

<div class="post-metadata">

**Author:** ![Karussell1](https://avatars.discourse-cdn.com/v4/letter/k/50afbb/32.png) [@Karussell1](https://discuss.elastic.co/u/Karussell1)\
**Post date:** [April 9, 2011, 2:38pm UTC](https://discuss.elastic.co/t/improve-query-speed/4180/11 "2011-04-09T14:38:44Z")

</div>

I'll investigate the index settings a bit more now:

[http://elasticsearch.karmi.cz/blog/2011/03/23/update-settings.html](http://elasticsearch.karmi.cz/blog/2011/03/23/update-settings.html)

E.g. decreasing mergeFactor should increase query speed (and slow down  
indexing)

On 7 Apr., 00:25, Karussell [tableyourt...@googlemail.com](mailto:tableyourt...@googlemail.com) wrote:

> On 4 Apr., 19:08, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> 
> > Scheduling queries every X seconds is good for now.
> 
> Thanks, ok.
> 
> What else would you suggest to tune query performance? Any cache  
> settings like there are in solr?
> 
> BTW: investigated facets and performance (after hacking) is now much  
> better 🙂

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [April 9, 2011, 10:13pm UTC](https://discuss.elastic.co/t/improve-query-speed/4180/12 "2011-04-09T22:13:33Z")

</div>

Yes, decreasing the merge factor will increase query speed, since less segments means less work when searching a single shard, but, it will make indexing slower / more expensive.

Another option to improve search speed is to reduce the term interval and play with the term divisor. Lowering the term interval will improve search perf, but, will cause more memory to be used (which you can control with the divisor). Both can also be set dynamically, though the term interval only applies for newly indexed docs.  
On Saturday, April 9, 2011 at 5:38 PM, Karussell wrote:

> I'll investigate the index settings a bit more now:
> 
> [http://elasticsearch.karmi.cz/blog/2011/03/23/update-settings.html](http://elasticsearch.karmi.cz/blog/2011/03/23/update-settings.html)
> 
> E.g. decreasing mergeFactor should increase query speed (and slow down  
> indexing)
> 
> On 7 Apr., 00:25, Karussell [tableyourt...@googlemail.com](mailto:tableyourt...@googlemail.com) wrote:
> 
> > On 4 Apr., 19:08, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> > 
> > > Scheduling queries every X seconds is good for now.
> > 
> > Thanks, ok.
> > 
> > What else would you suggest to tune query performance? Any cache  
> > settings like there are in solr?
> > 
> > BTW: investigated facets and performance (after hacking) is now much  
> > better 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:08am UTC](https://discuss.elastic.co/t/improve-query-speed/4180/13 "2017-07-06T04:08:39Z")

</div>