Cost of keeping non-searched indices?

Elastic Stack Elasticsearch
1

Hi,

What is the cost/overhead of simply keeping (i.e. not searching)
indices in an ES cluster?

For example:

  • Say you need to index news and decide to create monthly indices.
  • Say you want to keep even old news indexed, but don't want to search
    it all.
  • Say you want to be able to search only the last N months (i.e.
    indices) worth of news.

What is the cost associated with keeping older indices in the cluster?

Thanks,
Otis

Sematext is Hiring World-Wide -- http://sematext.com/about/jobs.html

2

Hi Otis,

you mean closed index as the open/close API defines?

--
Regards,
Lukas

On Tuesday, December 6, 2011 at 7:08 PM, Otis Gospodnetic wrote:

Hi,

What is the cost/overhead of simply keeping (i.e. not searching)
indices in an ES cluster?

For example:

  • Say you need to index news and decide to create monthly indices.
  • Say you want to keep even old news indexed, but don't want to search
    it all.
  • Say you want to be able to search only the last N months (i.e.
    indices) worth of news.

What is the cost associated with keeping older indices in the cluster?

Thanks,
Otis

Sematext is Hiring World-Wide -- Jobs - Sematext

3

Excellent, that's what I thought and hoped for!

Thanks Lukáš.

Related Qs about "A closed index can be opened which will then go
through the normal recovery process." from

  1. What is involved in index recovery? (I'm assuming a simple close
    followed by open results in this recovery process?) And roughly how
    long might that take, say for a 10GB index?

  2. What happens if some external indexer decides to a closed index? I
    assume an exception is thrown? Any way to get ES to open it, allow
    write, and maybe close if after the index has seen no reads/writes for
    N seconds/minutes?

Thanks,
Otis

Sematext is Hiring World-Wide -- Jobs - Sematext

On Dec 6, 2:58 pm, Lukáš Vlček lukas.vl...@gmail.com wrote:

Hi Otis,

you mean closed index as the open/close API defines?Elasticsearch Platform — Find real-time answers at scale | Elastic...

--
Regards,
Lukas

On Tuesday, December 6, 2011 at 7:08 PM, Otis Gospodnetic wrote:

Hi,

What is the cost/overhead of simply keeping (i.e. not searching)
indices in an ES cluster?

For example:

  • Say you need to index news and decide to create monthly indices.
  • Say you want to keep even old news indexed, but don't want to search
    it all.
  • Say you want to be able to search only the last N months (i.e.
    indices) worth of news.

What is the cost associated with keeping older indices in the cluster?

Thanks,
Otis

Sematext is Hiring World-Wide --Jobs - Sematext

4

On Wed, Dec 7, 2011 at 7:19 AM, Otis Gospodnetic <otis.gospodnetic@gmail.com

wrote:

Excellent, that's what I thought and hoped for!

Thanks Lukáš.

Related Qs about "A closed index can be opened which will then go
through the normal recovery process." from

Elasticsearch Platform — Find real-time answers at scale | Elastic

  1. What is involved in index recovery? (I'm assuming a simple close
    followed by open results in this recovery process?) And roughly how
    long might that take, say for a 10GB index?

It will be pretty fast, it mainly involves the cost of opening Lucene for
each shard, and replaying the transaction log.

  1. What happens if some external indexer decides to a closed index? I
    assume an exception is thrown? Any way to get ES to open it, allow
    write, and maybe close if after the index has seen no reads/writes for
    N seconds/minutes?

I assume you mean decides to index/search against hte index? An exception
will be thrown. You will need to manage opening / closing it yourself.

Thanks,
Otis

Sematext is Hiring World-Wide -- Jobs - Sematext

On Dec 6, 2:58 pm, Lukáš Vlček lukas.vl...@gmail.com wrote:

Hi Otis,

you mean closed index as the open/close API defines?
Elasticsearch Platform — Find real-time answers at scale | Elastic...

--
Regards,
Lukas

On Tuesday, December 6, 2011 at 7:08 PM, Otis Gospodnetic wrote:

Hi,

What is the cost/overhead of simply keeping (i.e. not searching)
indices in an ES cluster?

For example:

  • Say you need to index news and decide to create monthly indices.
  • Say you want to keep even old news indexed, but don't want to search
    it all.
  • Say you want to be able to search only the last N months (i.e.
    indices) worth of news.

What is the cost associated with keeping older indices in the cluster?

Thanks,
Otis

Sematext is Hiring World-Wide --Jobs - Sematext

5

On Wed, Dec 7, 2011 at 10:33 AM, Shay Banon kimchy@gmail.com wrote:

On Wed, Dec 7, 2011 at 7:19 AM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:

Excellent, that's what I thought and hoped for!

Thanks Lukáš.

Related Qs about "A closed index can be opened which will then go
through the normal recovery process." from

Elasticsearch Platform — Find real-time answers at scale | Elastic

  1. What is involved in index recovery? (I'm assuming a simple close
    followed by open results in this recovery process?) And roughly how
    long might that take, say for a 10GB index?

It will be pretty fast, it mainly involves the cost of opening Lucene for
each shard, and replaying the transaction log.

Probably worth noting that the transaction log step can be avoided if the
index is flushed after all docs have been added and prior to closing.

  1. What happens if some external indexer decides to a closed index? I
    assume an exception is thrown? Any way to get ES to open it, allow
    write, and maybe close if after the index has seen no reads/writes for
    N seconds/minutes?

I assume you mean decides to index/search against hte index? An exception
will be thrown. You will need to manage opening / closing it yourself.

Thanks,
Otis

Sematext is Hiring World-Wide -- Jobs - Sematext

On Dec 6, 2:58 pm, Lukáš Vlček lukas.vl...@gmail.com wrote:

Hi Otis,

you mean closed index as the open/close API defines?
Elasticsearch Platform — Find real-time answers at scale | Elastic...

--
Regards,
Lukas

On Tuesday, December 6, 2011 at 7:08 PM, Otis Gospodnetic wrote:

Hi,

What is the cost/overhead of simply keeping (i.e. not searching)
indices in an ES cluster?

For example:

  • Say you need to index news and decide to create monthly indices.
  • Say you want to keep even old news indexed, but don't want to search
    it all.
  • Say you want to be able to search only the last N months (i.e.
    indices) worth of news.

What is the cost associated with keeping older indices in the cluster?

Thanks,
Otis

Sematext is Hiring World-Wide --Jobs - Sematext

6

It will be pretty fast, it mainly involves the cost of opening Lucene for> > each shard, and replaying the transaction log.> > Probably worth noting that the transaction log step can be avoided if the> index is flushed after all docs have been added and prior to closing.

Aha, thanks, was going to ask - replaying transaction log for a day's
worth of transaction would probably not be fast enough, but flush
before close and avoiding xa log replaying sounds like what I'm
after. Thanks!

Otis

Sematext is Hiring World-Wide -- Jobs - Sematext

On Dec 7, 10:36 am, Michael Sick michael.s...@serenesoftware.com
wrote:

On Wed, Dec 7, 2011 at 10:33 AM, Shay Banon kim...@gmail.com wrote:

On Wed, Dec 7, 2011 at 7:19 AM, Otis Gospodnetic <
otis.gospodne...@gmail.com> wrote:

Excellent, that's what I thought and hoped for!

Thanks Lukáš.

Related Qs about "A closed index can be opened which will then go
through the normal recovery process." from

Elasticsearch Platform — Find real-time answers at scale | Elastic...

  1. What is involved in index recovery? (I'm assuming a simple close
    followed by open results in this recovery process?) And roughly how
    long might that take, say for a 10GB index?

It will be pretty fast, it mainly involves the cost of opening Lucene for
each shard, and replaying the transaction log.

Probably worth noting that the transaction log step can be avoided if the
index is flushed after all docs have been added and prior to closing.

  1. What happens if some external indexer decides to a closed index? I
    assume an exception is thrown? Any way to get ES to open it, allow
    write, and maybe close if after the index has seen no reads/writes for
    N seconds/minutes?

I assume you mean decides to index/search against hte index? An exception
will be thrown. You will need to manage opening / closing it yourself.

Thanks,
Otis

Sematext is Hiring World-Wide --Jobs - Sematext

On Dec 6, 2:58 pm, Lukáš Vlček lukas.vl...@gmail.com wrote:

Hi Otis,

you mean closed index as the open/close API defines?
Elasticsearch Platform — Find real-time answers at scale | Elastic...

--
Regards,
Lukas

On Tuesday, December 6, 2011 at 7:08 PM, Otis Gospodnetic wrote:

Hi,

What is the cost/overhead of simply keeping (i.e. not searching)
indices in an ES cluster?

For example:

  • Say you need to index news and decide to create monthly indices.
  • Say you want to keep even old news indexed, but don't want to search
    it all.
  • Say you want to be able to search only the last N months (i.e.
    indices) worth of news.

What is the cost associated with keeping older indices in the cluster?

Thanks,
Otis

Sematext is Hiring World-Wide --Jobs - Sematext