Multiple path.data and stripping

Elastic Stack Elasticsearch
1

All,

  1. If one adds additional devices post to a running server will es balance
    (move) the existing index to "even" out the drives (both space and io)?

  2. Can indexes be moved to a new drive?

Thanks

--

2

  1. When you add new devices, Elasticsearch will not move existing files to
    the additional device immediately, existing files will remain on the old
    devices. However, as new files are created Elasticsearch will create them
    on a device with the most available space, which means that all new files
    will be created on new devices until they fill up. So, basically, the
    process of creating new segments and merging old segments that
    elasticsearch goes through as it indexes new data will eventually balance
    things out.

  2. Yes, you can shut down node move indices to the new drive, replace
    path.data in config and start the node.

On Thursday, December 6, 2012 1:10:03 PM UTC-5, Kubes wrote:

All,

  1. If one adds additional devices post to a running server will es balance
    (move) the existing index to "even" out the drives (both space and io)?

  2. Can indexes be moved to a new drive?

Thanks

--

3

Thanks for the info.

A few more questions nn "moving" index

I have added the 2nd drive via the path.data. Now tt look like ES now has
created an directory for every index on the additional (new) drive. In
each has the shard directory (shard number) with the index, _state and
translog subdirectories. There is a state file.

So to move a index from initial drive (Drive A) to the new (Drive B), and I
just shutdown ES and move the file structure of that index, overwriting
what is on Drive B?

Thanks

On Fri, Dec 7, 2012 at 10:43 AM, Igor Motov imotov@gmail.com wrote:

  1. When you add new devices, Elasticsearch will not move existing files to
    the additional device immediately, existing files will remain on the old
    devices. However, as new files are created Elasticsearch will create them
    on a device with the most available space, which means that all new files
    will be created on new devices until they fill up. So, basically, the
    process of creating new segments and merging old segments that
    elasticsearch goes through as it indexes new data will eventually balance
    things out.

  2. Yes, you can shut down node move indices to the new drive, replace
    path.data in config and start the node.

On Thursday, December 6, 2012 1:10:03 PM UTC-5, Kubes wrote:

All,

  1. If one adds additional devices post to a running server will es
    balance (move) the existing index to "even" out the drives (both space and
    io)?

  2. Can indexes be moved to a new drive?

Thanks

--

--

4

Now, since you already started the node, it might be tricky. If you indexed
something after node was started, Elasticsearch autorefreshed some data, or
segment merge occurred it might have created some files in the new
directory. So, you cannot simply overwrite directories on drive B if they
have some files in them, you might loose some data. You need to carefully
"merge" them together to make sure that you don't loose any files (the
state-??? file will be in both structures, you can just keep one of them).

On Friday, December 7, 2012 11:28:32 AM UTC-5, Kubes wrote:

Thanks for the info.

A few more questions nn "moving" index

I have added the 2nd drive via the path.data. Now tt look like ES now has
created an directory for every index on the additional (new) drive. In
each has the shard directory (shard number) with the index, _state and
translog subdirectories. There is a state file.

So to move a index from initial drive (Drive A) to the new (Drive B), and
I just shutdown ES and move the file structure of that index, overwriting
what is on Drive B?

Thanks

On Fri, Dec 7, 2012 at 10:43 AM, Igor Motov <imo...@gmail.com<javascript:>

wrote:

  1. When you add new devices, Elasticsearch will not move existing files
    to the additional device immediately, existing files will remain on the old
    devices. However, as new files are created Elasticsearch will create them
    on a device with the most available space, which means that all new files
    will be created on new devices until they fill up. So, basically, the
    process of creating new segments and merging old segments that
    elasticsearch goes through as it indexes new data will eventually balance
    things out.

  2. Yes, you can shut down node move indices to the new drive, replace
    path.data in config and start the node.

On Thursday, December 6, 2012 1:10:03 PM UTC-5, Kubes wrote:

All,

  1. If one adds additional devices post to a running server will es
    balance (move) the existing index to "even" out the drives (both space and
    io)?

  2. Can indexes be moved to a new drive?

Thanks

--

--

5

Again Thansk.

The "good news" is that these are logstash indexes, which are by date and
are not written to post the date. So I think the move will still be okay,
as there is no "index/*" files for any shard.

But for future reference how does one "merge". (In ES or via the OS).
Could we export/import? (again how). Sorry these might newbie
questions....

On Fri, Dec 7, 2012 at 11:44 AM, Igor Motov imotov@gmail.com wrote:

Now, since you already started the node, it might be tricky. If you
indexed something after node was started, Elasticsearch autorefreshed some
data, or segment merge occurred it might have created some files in the
new directory. So, you cannot simply overwrite directories on drive B if
they have some files in them, you might loose some data. You need to
carefully "merge" them together to make sure that you don't loose any files
(the state-??? file will be in both structures, you can just keep one of
them).

On Friday, December 7, 2012 11:28:32 AM UTC-5, Kubes wrote:

Thanks for the info.

A few more questions nn "moving" index

I have added the 2nd drive via the path.data. Now tt look like ES now
has created an directory for every index on the additional (new) drive. In
each has the shard directory (shard number) with the index, _state and
translog subdirectories. There is a state file.

So to move a index from initial drive (Drive A) to the new (Drive B), and
I just shutdown ES and move the file structure of that index, overwriting
what is on Drive B?

Thanks

On Fri, Dec 7, 2012 at 10:43 AM, Igor Motov imo...@gmail.com wrote:

  1. When you add new devices, Elasticsearch will not move existing files
    to the additional device immediately, existing files will remain on the old
    devices. However, as new files are created Elasticsearch will create them
    on a device with the most available space, which means that all new files
    will be created on new devices until they fill up. So, basically, the
    process of creating new segments and merging old segments that
    elasticsearch goes through as it indexes new data will eventually balance
    things out.

  2. Yes, you can shut down node move indices to the new drive, replace
    path.data in config and start the node.

On Thursday, December 6, 2012 1:10:03 PM UTC-5, Kubes wrote:

All,

  1. If one adds additional devices post to a running server will es
    balance (move) the existing index to "even" out the drives (both space and
    io)?

  2. Can indexes be moved to a new drive?

Thanks

--

--

--

6

For the future reference, it's better not to merge :slight_smile: . Just don't start
the server after adding a new drive before you move all data you want to
move. There is really no support for merging, moving, exporting or
importing at the moment. It's all have to be done on OS level. The only
elasticsearch-based solution that I can think of is to reroute shards from
the node that you want to upgrade to other nodes, replace the drives and
then reroute shards back to this node.

On Friday, December 7, 2012 11:54:23 AM UTC-5, Kubes wrote:

Again Thansk.

The "good news" is that these are logstash indexes, which are by date and
are not written to post the date. So I think the move will still be okay,
as there is no "index/*" files for any shard.

But for future reference how does one "merge". (In ES or via the OS).
Could we export/import? (again how). Sorry these might newbie
questions....

On Fri, Dec 7, 2012 at 11:44 AM, Igor Motov <imo...@gmail.com<javascript:>

wrote:

Now, since you already started the node, it might be tricky. If you
indexed something after node was started, Elasticsearch autorefreshed some
data, or segment merge occurred it might have created some files in the
new directory. So, you cannot simply overwrite directories on drive B if
they have some files in them, you might loose some data. You need to
carefully "merge" them together to make sure that you don't loose any files
(the state-??? file will be in both structures, you can just keep one of
them).

On Friday, December 7, 2012 11:28:32 AM UTC-5, Kubes wrote:

Thanks for the info.

A few more questions nn "moving" index

I have added the 2nd drive via the path.data. Now tt look like ES now
has created an directory for every index on the additional (new) drive. In
each has the shard directory (shard number) with the index, _state and
translog subdirectories. There is a state file.

So to move a index from initial drive (Drive A) to the new (Drive B),
and I just shutdown ES and move the file structure of that index,
overwriting what is on Drive B?

Thanks

On Fri, Dec 7, 2012 at 10:43 AM, Igor Motov imo...@gmail.com wrote:

  1. When you add new devices, Elasticsearch will not move existing files
    to the additional device immediately, existing files will remain on the old
    devices. However, as new files are created Elasticsearch will create them
    on a device with the most available space, which means that all new files
    will be created on new devices until they fill up. So, basically, the
    process of creating new segments and merging old segments that
    elasticsearch goes through as it indexes new data will eventually balance
    things out.

  2. Yes, you can shut down node move indices to the new drive, replace
    path.data in config and start the node.

On Thursday, December 6, 2012 1:10:03 PM UTC-5, Kubes wrote:

All,

  1. If one adds additional devices post to a running server will es
    balance (move) the existing index to "even" out the drives (both space and
    io)?

  2. Can indexes be moved to a new drive?

Thanks

--

--

--

7

Perfect.
Thanks.

On Fri, Dec 7, 2012 at 12:13 PM, Igor Motov imotov@gmail.com wrote:

For the future reference, it's better not to merge :slight_smile: . Just don't start
the server after adding a new drive before you move all data you want to
move. There is really no support for merging, moving, exporting or
importing at the moment. It's all have to be done on OS level. The only
elasticsearch-based solution that I can think of is to reroute shards from
the node that you want to upgrade to other nodes, replace the drives and
then reroute shards back to this node.

On Friday, December 7, 2012 11:54:23 AM UTC-5, Kubes wrote:

Again Thansk.

The "good news" is that these are logstash indexes, which are by date and
are not written to post the date. So I think the move will still be okay,
as there is no "index/*" files for any shard.

But for future reference how does one "merge". (In ES or via the OS).
Could we export/import? (again how). Sorry these might newbie
questions....

On Fri, Dec 7, 2012 at 11:44 AM, Igor Motov imo...@gmail.com wrote:

Now, since you already started the node, it might be tricky. If you
indexed something after node was started, Elasticsearch autorefreshed some
data, or segment merge occurred it might have created some files in the
new directory. So, you cannot simply overwrite directories on drive B if
they have some files in them, you might loose some data. You need to
carefully "merge" them together to make sure that you don't loose any files
(the state-??? file will be in both structures, you can just keep one of
them).

On Friday, December 7, 2012 11:28:32 AM UTC-5, Kubes wrote:

Thanks for the info.

A few more questions nn "moving" index

I have added the 2nd drive via the path.data. Now tt look like ES now
has created an directory for every index on the additional (new) drive. In
each has the shard directory (shard number) with the index, _state and
translog subdirectories. There is a state file.

So to move a index from initial drive (Drive A) to the new (Drive B),
and I just shutdown ES and move the file structure of that index,
overwriting what is on Drive B?

Thanks

On Fri, Dec 7, 2012 at 10:43 AM, Igor Motov imo...@gmail.com wrote:

  1. When you add new devices, Elasticsearch will not move existing
    files to the additional device immediately, existing files will remain on
    the old devices. However, as new files are created Elasticsearch will
    create them on a device with the most available space, which means that all
    new files will be created on new devices until they fill up. So, basically,
    the process of creating new segments and merging old segments that
    elasticsearch goes through as it indexes new data will eventually balance
    things out.

  2. Yes, you can shut down node move indices to the new drive, replace
    path.data in config and start the node.

On Thursday, December 6, 2012 1:10:03 PM UTC-5, Kubes wrote:

All,

  1. If one adds additional devices post to a running server will es
    balance (move) the existing index to "even" out the drives (both space and
    io)?

  2. Can indexes be moved to a new drive?

Thanks

--

--

--

--