Backing up and Restoring a specific index

Elastic Stack Elasticsearch
1

Hi,

I have a multi-tenant system where each tenant has their own index. It
is sometimes required to restore a tenant to its previous state.

Is it possible to backup and restore each index (tenant) separately?

More specifically:

  1. Can I backup each index data folder in the gateway separately?
  2. Can I restore each index data folder to the gateway separately?
  3. What about the metadata? It holds data for all the indexes so when
    I restore a index folder it may be our of sync with the metadata.
  4. How do I empty the transaction log of data related to an index
    before I restore it? Do I need to flush? I wouldn't want to have data
    in the index that is both from the restored index and from the
    transaction log that was flushed after the restore.
  5. Can I somehow shutdown an index, and tell it to reload from the
    gateway (and ignore its work folder)?

Thanks,
Tal

2

No, there isn't support for per index management to this degree. Can be
implemented, with various difficulties depending on what exactly is needed.
For example, just snapshotting an index to a gateway, and then restoring
from it is simple. The metadata is the more complex part since it is not
stored on a per index leve, but globally.

-shay.banon

On Tue, Aug 10, 2010 at 6:05 PM, Tal talsalmona@gmail.com wrote:

Hi,

I have a multi-tenant system where each tenant has their own index. It
is sometimes required to restore a tenant to its previous state.

Is it possible to backup and restore each index (tenant) separately?

More specifically:

  1. Can I backup each index data folder in the gateway separately?
  2. Can I restore each index data folder to the gateway separately?
  3. What about the metadata? It holds data for all the indexes so when
    I restore a index folder it may be our of sync with the metadata.
  4. How do I empty the transaction log of data related to an index
    before I restore it? Do I need to flush? I wouldn't want to have data
    in the index that is both from the restored index and from the
    transaction log that was flushed after the restore.
  5. Can I somehow shutdown an index, and tell it to reload from the
    gateway (and ignore its work folder)?

Thanks,
Tal

3

Is this something you are planning for a future release?
I think that it is an important use case for multi-tenant
applications. In many cases you even let the customer (tenant owner or
admin) initiate the restore on their own.

Tal.

On Aug 10, 9:02 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

No, there isn't support for per index management to this degree. Can be
implemented, with various difficulties depending on what exactly is needed.
For example, just snapshotting an index to a gateway, and then restoring
from it is simple. The metadata is the more complex part since it is not
stored on a per index leve, but globally.

-shay.banon

On Tue, Aug 10, 2010 at 6:05 PM, Tal talsalm...@gmail.com wrote:

Hi,

I have a multi-tenant system where each tenant has their own index. It
is sometimes required to restore a tenant to its previous state.

Is it possible to backup and restore each index (tenant) separately?

More specifically:

  1. Can I backup each index data folder in the gateway separately?
  2. Can I restore each index data folder to the gateway separately?
  3. What about the metadata? It holds data for all the indexes so when
    I restore a index folder it may be our of sync with the metadata.
  4. How do I empty the transaction log of data related to an index
    before I restore it? Do I need to flush? I wouldn't want to have data
    in the index that is both from the restored index and from the
    transaction log that was flushed after the restore.
  5. Can I somehow shutdown an index, and tell it to reload from the
    gateway (and ignore its work folder)?

Thanks,
Tal

4

Agreed, it is important. Can you open an issue for this? I am currently
working on 0.9.1 and want to close what I plan, will try and schedule it for
the next version.

-shay.banon

On Wed, Aug 11, 2010 at 12:28 PM, Tal talsalmona@gmail.com wrote:

Is this something you are planning for a future release?
I think that it is an important use case for multi-tenant
applications. In many cases you even let the customer (tenant owner or
admin) initiate the restore on their own.

Tal.

On Aug 10, 9:02 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

No, there isn't support for per index management to this degree. Can be
implemented, with various difficulties depending on what exactly is
needed.
For example, just snapshotting an index to a gateway, and then restoring
from it is simple. The metadata is the more complex part since it is not
stored on a per index leve, but globally.

-shay.banon

On Tue, Aug 10, 2010 at 6:05 PM, Tal talsalm...@gmail.com wrote:

Hi,

I have a multi-tenant system where each tenant has their own index. It
is sometimes required to restore a tenant to its previous state.

Is it possible to backup and restore each index (tenant) separately?

More specifically:

  1. Can I backup each index data folder in the gateway separately?
  2. Can I restore each index data folder to the gateway separately?
  3. What about the metadata? It holds data for all the indexes so when
    I restore a index folder it may be our of sync with the metadata.
  4. How do I empty the transaction log of data related to an index
    before I restore it? Do I need to flush? I wouldn't want to have data
    in the index that is both from the restored index and from the
    transaction log that was flushed after the restore.
  5. Can I somehow shutdown an index, and tell it to reload from the
    gateway (and ignore its work folder)?

Thanks,
Tal

5

Yeah, I would like this feature, as well, but figured it wouldn't be
too hard to add myself outside of ES for my use case.

Thanks!

On Aug 11, 3:56 am, Shay Banon shay.ba...@elasticsearch.com wrote:

Agreed, it is important. Can you open an issue for this? I am currently
working on 0.9.1 and want to close what I plan, will try and schedule it for
the next version.

-shay.banon

On Wed, Aug 11, 2010 at 12:28 PM, Tal talsalm...@gmail.com wrote:

Is this something you are planning for a future release?
I think that it is an important use case for multi-tenant
applications. In many cases you even let the customer (tenant owner or
admin) initiate the restore on their own.

Tal.

On Aug 10, 9:02 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

No, there isn't support for per index management to this degree. Can be
implemented, with various difficulties depending on what exactly is
needed.
For example, just snapshotting an index to a gateway, and then restoring
from it is simple. The metadata is the more complex part since it is not
stored on a per index leve, but globally.

-shay.banon

On Tue, Aug 10, 2010 at 6:05 PM, Tal talsalm...@gmail.com wrote:

Hi,

I have a multi-tenant system where each tenant has their own index. It
is sometimes required to restore a tenant to its previous state.

Is it possible to backup and restore each index (tenant) separately?

More specifically:

  1. Can I backup each index data folder in the gateway separately?
  2. Can I restore each index data folder to the gateway separately?
  3. What about the metadata? It holds data for all the indexes so when
    I restore a index folder it may be our of sync with the metadata.
  4. How do I empty the transaction log of data related to an index
    before I restore it? Do I need to flush? I wouldn't want to have data
    in the index that is both from the restored index and from the
    transaction log that was flushed after the restore.
  5. Can I somehow shutdown an index, and tell it to reload from the
    gateway (and ignore its work folder)?

Thanks,
Tal

6

Added issue: Backing up and Restoring a Specific Index · Issue #313 · elastic/elasticsearch · GitHub

On Aug 11, 7:06 pm, Paul ppea...@gmail.com wrote:

Yeah, I would like this feature, as well, but figured it wouldn't be
too hard to add myself outside of ES for my use case.

Thanks!

On Aug 11, 3:56 am, Shay Banon shay.ba...@elasticsearch.com wrote:

Agreed, it is important. Can you open an issue for this? I am currently
working on 0.9.1 and want to close what I plan, will try and schedule it for
the next version.

-shay.banon

On Wed, Aug 11, 2010 at 12:28 PM, Tal talsalm...@gmail.com wrote:

Is this something you are planning for a future release?
I think that it is an important use case for multi-tenant
applications. In many cases you even let the customer (tenant owner or
admin) initiate the restore on their own.

Tal.

On Aug 10, 9:02 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

No, there isn't support for per index management to this degree. Can be
implemented, with various difficulties depending on what exactly is
needed.
For example, just snapshotting an index to a gateway, and then restoring
from it is simple. The metadata is the more complex part since it is not
stored on a per index leve, but globally.

-shay.banon

On Tue, Aug 10, 2010 at 6:05 PM, Tal talsalm...@gmail.com wrote:

Hi,

I have a multi-tenant system where each tenant has their own index. It
is sometimes required to restore a tenant to its previous state.

Is it possible to backup and restore each index (tenant) separately?

More specifically:

  1. Can I backup each index data folder in the gateway separately?
  2. Can I restore each index data folder to the gateway separately?
  3. What about the metadata? It holds data for all the indexes so when
    I restore a index folder it may be our of sync with the metadata.
  4. How do I empty the transaction log of data related to an index
    before I restore it? Do I need to flush? I wouldn't want to have data
    in the index that is both from the restored index and from the
    transaction log that was flushed after the restore.
  5. Can I somehow shutdown an index, and tell it to reload from the
    gateway (and ignore its work folder)?

Thanks,
Tal