Knapsack use case

Hi All,
I have some questions about the knapsack plugin [1].

My idea to use the tool to do a backup to a file, starting from a 0.90.x
instance and then restore it on a different 1.2.x or 1.3.x instance. I see
it can't be done directly, copying to a local/remote cluster.

Would it work doing an intermediate step with a file?
Or the backup still has metadata about the es version it was generated
from, making it impossible?

Is the snapshot and restore feature [2] useful in my use case, or not?

Is the knapsack plugin able to backup and restore also aliases and
mappings, or do I have to manually migrate them before restoring data?

Thanks for the patience and the great work!
Matteo

[1] https://github.com/jprante/elasticsearch-knapsack
[2]
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html

--
Matteo Moci
http://mox.fm

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAONgFZ60jWViqzRVO6_U-rYo6dUzunE3ojv%2BR5U8HX1Lwp4PdA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Snapshot/restore is always recommended, but is a 1.0 feature. This is a
standard API of ES and well supported by the ES team. With that, you can
handle all kinds of index data safely on a binary level, fully and
incrementally.

Knapsack plugin is for document export/import only. I wrote it to transport
_source data harvested over a long time period from a < 1.0 system to a
production system. It works on _source or stored fields only. It uses
search/query and bulk indexing API without snapshots, so it is up to the
admin to stop index writes while knapsack runs. There is also a lookup of
index settings and mappings, this information is also included in the
export archive file, and re-applied at import time. But, there is no check
if these settings/mappings can be applied on the target successfully, this
is left to the admin to prepare plugins, analyzers, etc. Aliases are not
transported but this is a good idea for improvement.

Currently, knapsack plugin does not work on ES 1.3 but I am progressing to
implement this. I am adding a Java-level API. Currently it is REST only.

Jörg

On Thu, Jul 31, 2014 at 11:05 AM, Matteo Moci mox601@gmail.com wrote:

Hi All,
I have some questions about the knapsack plugin [1].

My idea to use the tool to do a backup to a file, starting from a 0.90.x
instance and then restore it on a different 1.2.x or 1.3.x instance. I see
it can't be done directly, copying to a local/remote cluster.

Would it work doing an intermediate step with a file?
Or the backup still has metadata about the es version it was generated
from, making it impossible?

Is the snapshot and restore feature [2] useful in my use case, or not?

Is the knapsack plugin able to backup and restore also aliases and
mappings, or do I have to manually migrate them before restoring data?

Thanks for the patience and the great work!
Matteo

[1] https://github.com/jprante/elasticsearch-knapsack
[2]
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html

--
Matteo Moci
http://mox.fm

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAONgFZ60jWViqzRVO6_U-rYo6dUzunE3ojv%2BR5U8HX1Lwp4PdA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAONgFZ60jWViqzRVO6_U-rYo6dUzunE3ojv%2BR5U8HX1Lwp4PdA%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGrXthJjCchEf2oyvXKnSZyBp31nvnAeXwAZJaEkvnT5Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

…just a small update in case anyone was wondering:

I completed an elasticsearch-knapsack export to file from a 0.90.7 (with
the plugin built with 0.90.7 dependencies) that was correctly re-imported
in a 1.4.2 instance with the latest plugin version, including settings and
aliases.

I just checked out the source from github and changed 2-3 lines due to API
changes and assembled the plugin to be installed on a 0.90.7 instance.

Just wanted to say thanks!

Best,
Matteo

On Thu, Jul 31, 2014 at 11:33 AM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

Snapshot/restore is always recommended, but is a 1.0 feature. This is a
standard API of ES and well supported by the ES team. With that, you can
handle all kinds of index data safely on a binary level, fully and
incrementally.

Knapsack plugin is for document export/import only. I wrote it to
transport _source data harvested over a long time period from a < 1.0
system to a production system. It works on _source or stored fields only.
It uses search/query and bulk indexing API without snapshots, so it is up
to the admin to stop index writes while knapsack runs. There is also a
lookup of index settings and mappings, this information is also included in
the export archive file, and re-applied at import time. But, there is no
check if these settings/mappings can be applied on the target successfully,
this is left to the admin to prepare plugins, analyzers, etc. Aliases are
not transported but this is a good idea for improvement.

Currently, knapsack plugin does not work on ES 1.3 but I am progressing to
implement this. I am adding a Java-level API. Currently it is REST only.

Jörg

On Thu, Jul 31, 2014 at 11:05 AM, Matteo Moci mox601@gmail.com wrote:

Hi All,
I have some questions about the knapsack plugin [1].

My idea to use the tool to do a backup to a file, starting from a 0.90.x
instance and then restore it on a different 1.2.x or 1.3.x instance. I see
it can't be done directly, copying to a local/remote cluster.

Would it work doing an intermediate step with a file?
Or the backup still has metadata about the es version it was generated
from, making it impossible?

Is the snapshot and restore feature [2] useful in my use case, or not?

Is the knapsack plugin able to backup and restore also aliases and
mappings, or do I have to manually migrate them before restoring data?

Thanks for the patience and the great work!
Matteo

[1] https://github.com/jprante/elasticsearch-knapsack
[2]
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html

--
Matteo Moci
http://mox.fm

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAONgFZ60jWViqzRVO6_U-rYo6dUzunE3ojv%2BR5U8HX1Lwp4PdA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAONgFZ60jWViqzRVO6_U-rYo6dUzunE3ojv%2BR5U8HX1Lwp4PdA%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGrXthJjCchEf2oyvXKnSZyBp31nvnAeXwAZJaEkvnT5Q%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGrXthJjCchEf2oyvXKnSZyBp31nvnAeXwAZJaEkvnT5Q%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
Matteo Moci
http://mox.fm

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAONgFZ6PvJyeF04ERyeb26LhXoHR%3DMMs5sc5KV2ASLF_UK6b%3Dw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.