# A lightweight partial index
**URL:** https://discuss.elastic.co/t/a-lightweight-partial-index/11746
**Category:** Elasticsearch
**Created:** [April 30, 2013, 10:18am UTC](https://discuss.elastic.co/t/a-lightweight-partial-index/11746 "2013-04-30T10:18:02Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@Jon\_Pither](https://discuss.elastic.co/u/Jon_Pither)
#### Post date: [April 30, 2013, 10:18am UTC](https://discuss.elastic.co/t/a-lightweight-partial-index/11746/1 "2013-04-30T10:18:02Z")
Hi,
We have an ES index of around 12.5 gb and growing. We use the index in
various ways.
For one use case we hit the index with an 'ids' query - passing around 500
ids typically and then pulling the documents back. The documents are fairly
large and we don't need all the data in them.
Performance is extremely important here, and so I did a test where I hit a
lightweight version of the same index (I re-indexed everything minus all
the fields we don't need into a new index), and I got 3x speed increase.
The new index is around 2 gb.
Obviously this creates complexity because now we have 2 indexes - a
lightweight one and a 'full fat' one.
I'm wondering if anyone has gone with this approach before - is it a
good/bad idea? And it would be great if ES has any smarts and plugins that
would allow us to index into the full-fat one, and have the other
lightweight one indexed also (with just the fields we want).
BTW I did try just fetching back the fields we care about from the full-fat
one rather than having a completely separate index, but this didn't seem to
give us much gain,
thanks,
Jon.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@btiernay](https://discuss.elastic.co/u/btiernay)
#### Post date: [April 30, 2013, 11:49am UTC](https://discuss.elastic.co/t/a-lightweight-partial-index/11746/2 "2013-04-30T11:49:10Z")
Have you considered using partial\_fields[http://www.elasticsearch.org/guide/reference/api/search/fields/](http://www.elasticsearch.org/guide/reference/api/search/fields/) in
your queries? This will lighten the network burden.
On Tuesday, 30 April 2013 06:18:02 UTC-4, Jon Pither wrote:
> Hi,
>
> We have an ES index of around 12.5 gb and growing. We use the index in
> various ways.
>
> For one use case we hit the index with an 'ids' query - passing around 500
> ids typically and then pulling the documents back. The documents are fairly
> large and we don't need all the data in them.
>
> Performance is extremely important here, and so I did a test where I hit a
> lightweight version of the same index (I re-indexed everything minus all
> the fields we don't need into a new index), and I got 3x speed increase.
> The new index is around 2 gb.
>
> Obviously this creates complexity because now we have 2 indexes - a
> lightweight one and a 'full fat' one.
>
> I'm wondering if anyone has gone with this approach before - is it a
> good/bad idea? And it would be great if ES has any smarts and plugins that
> would allow us to index into the full-fat one, and have the other
> lightweight one indexed also (with just the fields we want).
>
> BTW I did try just fetching back the fields we care about from the
> full-fat one rather than having a completely separate index, but this
> didn't seem to give us much gain,
>
> thanks,
>
> Jon.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)
#### Post date: [April 30, 2013, 7:48pm UTC](https://discuss.elastic.co/t/a-lightweight-partial-index/11746/3 "2013-04-30T19:48:49Z")
One thing to note is that the ìds`filter is not cached by default, but using a`terms`filter on the`\_id` field IS cached, so you may want to try
that instead.
clint
On Tue, Apr 30, 2013 at 1:49 PM, btiernay [rtiernay@gmail.com](mailto:rtiernay@gmail.com) wrote:
> Have you considered using partial\_fields[http://www.elasticsearch.org/guide/reference/api/search/fields/](http://www.elasticsearch.org/guide/reference/api/search/fields/) in
> your queries? This will lighten the network burden.
>
> On Tuesday, 30 April 2013 06:18:02 UTC-4, Jon Pither wrote:
>
> > Hi,
> >
> > We have an ES index of around 12.5 gb and growing. We use the index in
> > various ways.
> >
> > For one use case we hit the index with an 'ids' query - passing around
> > 500 ids typically and then pulling the documents back. The documents are
> > fairly large and we don't need all the data in them.
> >
> > Performance is extremely important here, and so I did a test where I hit
> > a lightweight version of the same index (I re-indexed everything minus all
> > the fields we don't need into a new index), and I got 3x speed increase.
> > The new index is around 2 gb.
> >
> > Obviously this creates complexity because now we have 2 indexes - a
> > lightweight one and a 'full fat' one.
> >
> > I'm wondering if anyone has gone with this approach before - is it a
> > good/bad idea? And it would be great if ES has any smarts and plugins that
> > would allow us to index into the full-fat one, and have the other
> > lightweight one indexed also (with just the fields we want).
> >
> > BTW I did try just fetching back the fields we care about from the
> > full-fat one rather than having a completely separate index, but this
> > didn't seem to give us much gain,
> >
> > thanks,
> >
> > Jon.
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@taras](https://discuss.elastic.co/u/taras)
#### Post date: [April 30, 2013, 11:36pm UTC](https://discuss.elastic.co/t/a-lightweight-partial-index/11746/4 "2013-04-30T23:36:23Z")
For our application, I hand-select what fields get indexed and what get
stored in \_source. Depending on your application, you are not likely to
need every field to query on, and not every field need to be retrieved.
There also need not be overlap between the two. So you can index fields
a,d,j and retrieve fields a,b,c.
There is also a difference between storing content in \_source[http://www.elasticsearch.org/guide/reference/mapping/source-field/](http://www.elasticsearch.org/guide/reference/mapping/source-field/)and using "store":"yes" for persistence. store is good for retrieving
individual fields, and if you need to get multiple fields you're probably
better off with just relying of \_source.
With some careful balancing of what goes where we are reducing 100GB of raw
data into 2GB of index.
On Tuesday, April 30, 2013 12:48:49 PM UTC-7, Clinton Gormley wrote:
> One thing to note is that the ìds`filter is not cached by default, but using a`terms`filter on the`\_id` field IS cached, so you may want to try
> that instead.
>
> clint
>
> On Tue, Apr 30, 2013 at 1:49 PM, btiernay \<[rtie...@gmail.com](mailto:rtie...@gmail.com) \
>
> > wrote:
>
> > Have you considered using partial\_fields[http://www.elasticsearch.org/guide/reference/api/search/fields/](http://www.elasticsearch.org/guide/reference/api/search/fields/) in
> > your queries? This will lighten the network burden.
> >
> > On Tuesday, 30 April 2013 06:18:02 UTC-4, Jon Pither wrote:
> >
> > > Hi,
> > >
> > > We have an ES index of around 12.5 gb and growing. We use the index in
> > > various ways.
> > >
> > > For one use case we hit the index with an 'ids' query - passing around
> > > 500 ids typically and then pulling the documents back. The documents are
> > > fairly large and we don't need all the data in them.
> > >
> > > Performance is extremely important here, and so I did a test where I hit
> > > a lightweight version of the same index (I re-indexed everything minus all
> > > the fields we don't need into a new index), and I got 3x speed increase.
> > > The new index is around 2 gb.
> > >
> > > Obviously this creates complexity because now we have 2 indexes - a
> > > lightweight one and a 'full fat' one.
> > >
> > > I'm wondering if anyone has gone with this approach before - is it a
> > > good/bad idea? And it would be great if ES has any smarts and plugins that
> > > would allow us to index into the full-fat one, and have the other
> > > lightweight one indexed also (with just the fields we want).
> > >
> > > BTW I did try just fetching back the fields we care about from the
> > > full-fat one rather than having a completely separate index, but this
> > > didn't seem to give us much gain,
> > >
> > > thanks,
> > >
> > > Jon.
> > >
> > > --
> > > You received this message because you are subscribed to the Google Groups
> > > "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send an
> > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 2:38am UTC](https://discuss.elastic.co/t/a-lightweight-partial-index/11746/5 "2017-07-06T02:38:58Z")