# Increasing performance with idle resources
**URL:** https://discuss.elastic.co/t/increasing-performance-with-idle-resources/11637
**Category:** Elasticsearch
**Created:** [April 18, 2013, 11:33pm UTC](https://discuss.elastic.co/t/increasing-performance-with-idle-resources/11637 "2013-04-18T23:33:13Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@Luke\_McCarthy](https://discuss.elastic.co/u/Luke_McCarthy)
#### Post date: [April 18, 2013, 11:33pm UTC](https://discuss.elastic.co/t/increasing-performance-with-idle-resources/11637/1 "2013-04-18T23:33:13Z")
Hi all,
I'm running elasticsearch with a single node on a machine with 24 total
cores (4x6) and 48GB RAM. The searches I'll be talking about here are run
over a pair of indices, each with around 2 million documents (~60GB data).
Each index has 5 shards and 0 replicas. The indices are continuously
updated through the bulk API with a few hundred new documents every second.
When the node is idle (aside from the bulk indexing), Lucene query\_string
queries over these indices can take \>2s to complete. Using bigdesk, I can
see plenty of memory and CPU unused during these searches.
Are there particular elasticsearch settings I can look at to try and better
utilize the idle resources to complete these searches more quickly? Should
I consider running multiple nodes on this single machine?
Cheers,
Luke
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@ppearcy](https://discuss.elastic.co/u/ppearcy)
#### Post date: [April 19, 2013, 12:48am UTC](https://discuss.elastic.co/t/increasing-performance-with-idle-resources/11637/2 "2013-04-19T00:48:35Z")
Hi Luke,
That sounds like a pretty heavy stream of updates, basically re-indexing
all your content every 2 hours.
I am guessing that you are disk bound.
How much memory is allocated to the ES heap? You probably want half
reserved for ES and half reserved for the system.
Increasing the translog flush threshold parameters may help to reduce the
commits that happen to the segments.
You may also see a benefit keeping the shard count around the number of
cores of the system.
Using filters on the search side where possible will also speed things up,
since they are cached.
There are likely other approaches I'm not aware of.
Best Regards,
Paul
On Thursday, April 18, 2013 5:33:13 PM UTC-6, Luke McCarthy wrote:
> Hi all,
>
> I'm running elasticsearch with a single node on a machine with 24 total
> cores (4x6) and 48GB RAM. The searches I'll be talking about here are run
> over a pair of indices, each with around 2 million documents (~60GB data).
> Each index has 5 shards and 0 replicas. The indices are continuously
> updated through the bulk API with a few hundred new documents every second.
>
> When the node is idle (aside from the bulk indexing), Lucene query\_string
> queries over these indices can take \>2s to complete. Using bigdesk, I can
> see plenty of memory and CPU unused during these searches.
>
> Are there particular elasticsearch settings I can look at to try and
> better utilize the idle resources to complete these searches more quickly?
> Should I consider running multiple nodes on this single machine?
>
> Cheers,
>
> Luke
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Luke\_McCarthy](https://discuss.elastic.co/u/Luke_McCarthy)
#### Post date: [April 19, 2013, 12:54am UTC](https://discuss.elastic.co/t/increasing-performance-with-idle-resources/11637/3 "2013-04-19T00:54:44Z")
On Thu, Apr 18, 2013 at 5:48 PM, ppearcy [ppearcy@gmail.com](mailto:ppearcy@gmail.com) wrote:
> Hi Luke,
> That sounds like a pretty heavy stream of updates, basically re-indexing
> all your content every 2 hours.
>
> I am guessing that you are disk bound.
Apparently not, according to bigdesk and iostat.
> How much memory is allocated to the ES heap? You probably want half
> reserved for ES and half reserved for the system.
Half (24gb) is allocated to heap, but not all of it is being used most of
the time (again, according to bigdesk).
Increasing the translog flush threshold parameters may help to reduce the
> commits that happen to the segments.
Assuming that this will reduce the availability of the data, I will not be
able to do this (I am expected to return things that were indexed a few
seconds ago…)
You may also see a benefit keeping the shard count around the number of
> cores of the system.
Shard count? Or segment count? I was under the impression that increasing
either of these would actually reduce search performance.
> Using filters on the search side where possible will also speed things up,
> since they are cached.
Yes, filtered queries are fine. It's only query\_string queries that I'd
like to improve performance for (note that wrapping the query\_string query
in a filter doesn't seem to improve anything…)
Thanks for your input,
Luke
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@ppearcy](https://discuss.elastic.co/u/ppearcy)
#### Post date: [April 19, 2013, 2:44am UTC](https://discuss.elastic.co/t/increasing-performance-with-idle-resources/11637/4 "2013-04-19T02:44:44Z")
Translog doesn't effect availability of data (it is different from a
commit), instead it causes fewer merges, at the cost of slower recovery
times during a restart. You should also look at various merge policy
settings and try to tune them for a high index load.
I believe there is a search thread per shard not segment (I could be
wrong), so I do think increasing your shard count will help in this case
due to the high volume of updates, that's what I'd try.
Best Regards,
Paul
On Thursday, April 18, 2013 6:54:44 PM UTC-6, Luke McCarthy wrote:
> On Thu, Apr 18, 2013 at 5:48 PM, ppearcy \<[ppe...@gmail.com](mailto:ppe...@gmail.com) \\>wrote:
>
> > Hi Luke,
> > That sounds like a pretty heavy stream of updates, basically
> > re-indexing all your content every 2 hours.
> >
> > I am guessing that you are disk bound.
>
> Apparently not, according to bigdesk and iostat.
>
> > How much memory is allocated to the ES heap? You probably want half
> > reserved for ES and half reserved for the system.
>
> Half (24gb) is allocated to heap, but not all of it is being used most of
> the time (again, according to bigdesk).
>
> Increasing the translog flush threshold parameters may help to reduce the
>
> > commits that happen to the segments.
>
> Assuming that this will reduce the availability of the data, I will not be
> able to do this (I am expected to return things that were indexed a few
> seconds ago…)
>
> You may also see a benefit keeping the shard count around the number of
>
> > cores of the system.
>
> Shard count? Or segment count? I was under the impression that increasing
> either of these would actually reduce search performance.
>
> > Using filters on the search side where possible will also speed things
> > up, since they are cached.
>
> Yes, filtered queries are fine. It's only query\_string queries that I'd
> like to improve performance for (note that wrapping the query\_string query
> in a filter doesn't seem to improve anything…)
>
> Thanks for your input,
>
> Luke
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 2:40am UTC](https://discuss.elastic.co/t/increasing-performance-with-idle-resources/11637/5 "2017-07-06T02:40:31Z")