# Cost of keeping non-searched indices?
**URL:** https://discuss.elastic.co/t/cost-of-keeping-non-searched-indices/6080
**Category:** Elasticsearch
**Created:** [December 6, 2011, 6:08pm UTC](https://discuss.elastic.co/t/cost-of-keeping-non-searched-indices/6080 "2011-12-06T18:08:58Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@otisg](https://discuss.elastic.co/u/otisg)
#### Post date: [December 6, 2011, 6:08pm UTC](https://discuss.elastic.co/t/cost-of-keeping-non-searched-indices/6080/1 "2011-12-06T18:08:58Z")
Hi,
What is the cost/overhead of simply keeping (i.e. not searching)
indices in an ES cluster?
For example:
- Say you need to index news and decide to create monthly indices.
- Say you want to keep even old news indexed, but don't want to search
it all.
- Say you want to be able to search only the last N months (i.e.
indices) worth of news.
What is the cost associated with keeping older indices in the cluster?
## Thanks, Otis
Sematext is Hiring World-Wide -- [http://sematext.com/about/jobs.html](http://sematext.com/about/jobs.html)
---
### Author:  [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)
#### Post date: [December 6, 2011, 7:58pm UTC](https://discuss.elastic.co/t/cost-of-keeping-non-searched-indices/6080/2 "2011-12-06T19:58:42Z")
Hi Otis,
you mean closed index as the open/close API defines?
> **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)**
>
> Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today.
--
Regards,
Lukas
On Tuesday, December 6, 2011 at 7:08 PM, Otis Gospodnetic wrote:
> Hi,
>
> What is the cost/overhead of simply keeping (i.e. not searching)
> indices in an ES cluster?
>
> For example:
>
> - Say you need to index news and decide to create monthly indices.
> - Say you want to keep even old news indexed, but don't want to search
> it all.
> - Say you want to be able to search only the last N months (i.e.
> indices) worth of news.
>
> What is the cost associated with keeping older indices in the cluster?
>
> ## Thanks, Otis
>
> Sematext is Hiring World-Wide -- [Jobs - Sematext](http://sematext.com/about/jobs.html)
---
### Author:  [@otisg](https://discuss.elastic.co/u/otisg)
#### Post date: [December 7, 2011, 5:19am UTC](https://discuss.elastic.co/t/cost-of-keeping-non-searched-indices/6080/3 "2011-12-07T05:19:49Z")
Excellent, that's what I thought and hoped for!
Thanks Lukáš.
Related Qs about "A closed index can be opened which will then go
through the normal recovery process." from
> **[Elastic — The Search AI Company](https://www.elastic.co)**
>
> Power insights and outcomes with The Elastic Search AI Platform. See into your data and find answers that matter with enterprise solutions designed to help you accelerate time to insight. Try Elastic ...
1. What is involved in index recovery? (I'm assuming a simple close
followed by open results in this recovery process?) And roughly how
long might that take, say for a 10GB index?
2. What happens if some external indexer decides to a closed index? I
assume an exception is thrown? Any way to get ES to open it, allow
write, and maybe close if after the index has seen no reads/writes for
N seconds/minutes?
## Thanks, Otis
Sematext is Hiring World-Wide -- [Jobs](http://sematext.com/about/jobs.html)
On Dec 6, 2:58 pm, Lukáš Vlček [lukas.vl...@gmail.com](mailto:lukas.vl...@gmail.com) wrote:
> Hi Otis,
>
> you mean closed index as the open/close API defines?[Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/guide/reference/api/admin-indices-open-c)...
>
> --
> Regards,
> Lukas
>
> On Tuesday, December 6, 2011 at 7:08 PM, Otis Gospodnetic wrote:
>
> > Hi,
>
> > What is the cost/overhead of simply keeping (i.e. not searching)
> > indices in an ES cluster?
>
> > For example:
> >
> > - Say you need to index news and decide to create monthly indices.
> > - Say you want to keep even old news indexed, but don't want to search
> > it all.
> > - Say you want to be able to search only the last N months (i.e.
> > indices) worth of news.
>
> > What is the cost associated with keeping older indices in the cluster?
>
> > ## Thanks, Otis
> >
> > Sematext is Hiring World-Wide --[Jobs](http://sematext.com/about/jobs.html)
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [December 7, 2011, 3:33pm UTC](https://discuss.elastic.co/t/cost-of-keeping-non-searched-indices/6080/4 "2011-12-07T15:33:03Z")
On Wed, Dec 7, 2011 at 7:19 AM, Otis Gospodnetic \<[otis.gospodnetic@gmail.com](mailto:otis.gospodnetic@gmail.com)
> wrote:
> Excellent, that's what I thought and hoped for!
>
> Thanks Lukáš.
>
> Related Qs about "A closed index can be opened which will then go
> through the normal recovery process." from
>
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/admin-indices-open-close.html)
>
> 1. What is involved in index recovery? (I'm assuming a simple close
> followed by open results in this recovery process?) And roughly how
> long might that take, say for a 10GB index?
It will be pretty fast, it mainly involves the cost of opening Lucene for
each shard, and replaying the transaction log.
> 1. What happens if some external indexer decides to a closed index? I
> assume an exception is thrown? Any way to get ES to open it, allow
> write, and maybe close if after the index has seen no reads/writes for
> N seconds/minutes?
I assume you mean decides to index/search against hte index? An exception
will be thrown. You will need to manage opening / closing it yourself.
> ## Thanks, Otis
>
> Sematext is Hiring World-Wide -- [Jobs - Sematext](http://sematext.com/about/jobs.html)
>
> On Dec 6, 2:58 pm, Lukáš Vlček [lukas.vl...@gmail.com](mailto:lukas.vl...@gmail.com) wrote:
>
> > Hi Otis,
> >
> > you mean closed index as the open/close API defines?
> > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/admin-indices-open-c)...
> >
> > --
> > Regards,
> > Lukas
> >
> > On Tuesday, December 6, 2011 at 7:08 PM, Otis Gospodnetic wrote:
> >
> > > Hi,
> >
> > > What is the cost/overhead of simply keeping (i.e. not searching)
> > > indices in an ES cluster?
> >
> > > For example:
> > >
> > > - Say you need to index news and decide to create monthly indices.
> > > - Say you want to keep even old news indexed, but don't want to search
> > > it all.
> > > - Say you want to be able to search only the last N months (i.e.
> > > indices) worth of news.
> >
> > > What is the cost associated with keeping older indices in the cluster?
> >
> > > ## Thanks, Otis
> > >
> > > Sematext is Hiring World-Wide --[Jobs - Sematext](http://sematext.com/about/jobs.html)
---
### Author:  [@Michael\_Sick](https://discuss.elastic.co/u/Michael_Sick)
#### Post date: [December 7, 2011, 3:36pm UTC](https://discuss.elastic.co/t/cost-of-keeping-non-searched-indices/6080/5 "2011-12-07T15:36:14Z")
On Wed, Dec 7, 2011 at 10:33 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote:
> On Wed, Dec 7, 2011 at 7:19 AM, Otis Gospodnetic \<
> [otis.gospodnetic@gmail.com](mailto:otis.gospodnetic@gmail.com)\> wrote:
>
> > Excellent, that's what I thought and hoped for!
> >
> > Thanks Lukáš.
> >
> > Related Qs about "A closed index can be opened which will then go
> > through the normal recovery process." from
> >
> > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/admin-indices-open-close.html)
> >
> > 1. What is involved in index recovery? (I'm assuming a simple close
> > followed by open results in this recovery process?) And roughly how
> > long might that take, say for a 10GB index?
>
> It will be pretty fast, it mainly involves the cost of opening Lucene for
> each shard, and replaying the transaction log.
Probably worth noting that the transaction log step can be avoided if the
index is flushed after all docs have been added and prior to closing.
> > 1. What happens if some external indexer decides to a closed index? I
> > assume an exception is thrown? Any way to get ES to open it, allow
> > write, and maybe close if after the index has seen no reads/writes for
> > N seconds/minutes?
>
> I assume you mean decides to index/search against hte index? An exception
> will be thrown. You will need to manage opening / closing it yourself.
>
> > ## Thanks, Otis
> >
> > Sematext is Hiring World-Wide -- [Jobs - Sematext](http://sematext.com/about/jobs.html)
> >
> > On Dec 6, 2:58 pm, Lukáš Vlček [lukas.vl...@gmail.com](mailto:lukas.vl...@gmail.com) wrote:
> >
> > > Hi Otis,
> > >
> > > you mean closed index as the open/close API defines?
> > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/admin-indices-open-c)...
> > >
> > > --
> > > Regards,
> > > Lukas
> > >
> > > On Tuesday, December 6, 2011 at 7:08 PM, Otis Gospodnetic wrote:
> > >
> > > > Hi,
> > >
> > > > What is the cost/overhead of simply keeping (i.e. not searching)
> > > > indices in an ES cluster?
> > >
> > > > For example:
> > > >
> > > > - Say you need to index news and decide to create monthly indices.
> > > > - Say you want to keep even old news indexed, but don't want to search
> > > > it all.
> > > > - Say you want to be able to search only the last N months (i.e.
> > > > indices) worth of news.
> > >
> > > > What is the cost associated with keeping older indices in the cluster?
> > >
> > > > ## Thanks, Otis
> > > >
> > > > Sematext is Hiring World-Wide --[Jobs - Sematext](http://sematext.com/about/jobs.html)
---
### Author:  [@otisg](https://discuss.elastic.co/u/otisg)
#### Post date: [December 7, 2011, 9:59pm UTC](https://discuss.elastic.co/t/cost-of-keeping-non-searched-indices/6080/6 "2011-12-07T21:59:46Z")
> > It will be pretty fast, it mainly involves the cost of opening Lucene for\> \> each shard, and replaying the transaction log.\> \> Probably worth noting that the transaction log step can be avoided if the\> index is flushed after all docs have been added and prior to closing.
Aha, thanks, was going to ask - replaying transaction log for a day's
worth of transaction would probably not be fast enough, but flush
before close and avoiding xa log replaying sounds like what I'm
after. Thanks!
## Otis
Sematext is Hiring World-Wide -- [Jobs](http://sematext.com/about/jobs.html)
On Dec 7, 10:36 am, Michael Sick [michael.s...@serenesoftware.com](mailto:michael.s...@serenesoftware.com)
wrote:
> On Wed, Dec 7, 2011 at 10:33 AM, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
>
> > On Wed, Dec 7, 2011 at 7:19 AM, Otis Gospodnetic \<
> > [otis.gospodne...@gmail.com](mailto:otis.gospodne...@gmail.com)\> wrote:
>
> > > Excellent, that's what I thought and hoped for!
>
> > > Thanks Lukáš.
>
> > > Related Qs about "A closed index can be opened which will then go
> > > through the normal recovery process." from
>
> > > [Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/guide/reference/api/admin-indices-open-c)...
>
> > > 1. What is involved in index recovery? (I'm assuming a simple close
> > > followed by open results in this recovery process?) And roughly how
> > > long might that take, say for a 10GB index?
>
> > It will be pretty fast, it mainly involves the cost of opening Lucene for
> > each shard, and replaying the transaction log.
>
> Probably worth noting that the transaction log step can be avoided if the
> index is flushed after all docs have been added and prior to closing.
>
> > > 1. What happens if some external indexer decides to a closed index? I
> > > assume an exception is thrown? Any way to get ES to open it, allow
> > > write, and maybe close if after the index has seen no reads/writes for
> > > N seconds/minutes?
>
> > I assume you mean decides to index/search against hte index? An exception
> > will be thrown. You will need to manage opening / closing it yourself.
>
> > > ## Thanks, Otis
> > >
> > > Sematext is Hiring World-Wide --[Jobs](http://sematext.com/about/jobs.html)
>
> > > On Dec 6, 2:58 pm, Lukáš Vlček [lukas.vl...@gmail.com](mailto:lukas.vl...@gmail.com) wrote:
> > >
> > > > Hi Otis,
>
> > > > you mean closed index as the open/close API defines?
> > > > [Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/guide/reference/api/admin-indices-open-c)...
>
> > > > --
> > > > Regards,
> > > > Lukas
>
> > > > On Tuesday, December 6, 2011 at 7:08 PM, Otis Gospodnetic wrote:
> > > >
> > > > > Hi,
>
> > > > > What is the cost/overhead of simply keeping (i.e. not searching)
> > > > > indices in an ES cluster?
>
> > > > > For example:
> > > > >
> > > > > - Say you need to index news and decide to create monthly indices.
> > > > > - Say you want to keep even old news indexed, but don't want to search
> > > > > it all.
> > > > > - Say you want to be able to search only the last N months (i.e.
> > > > > indices) worth of news.
>
> > > > > What is the cost associated with keeping older indices in the cluster?
>
> > > > > ## Thanks, Otis
> > > > >
> > > > > Sematext is Hiring World-Wide --[Jobs](http://sematext.com/about/jobs.html)
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 3:46am UTC](https://discuss.elastic.co/t/cost-of-keeping-non-searched-indices/6080/7 "2017-07-06T03:46:05Z")