Clearing the gateway on 0.10.0

Elastic Stack Elasticsearch
1

Still in dev mode, so am clearing my data semi-frequently.

I noticed that on 0.10.0, when I clear the gateway, nodes still come
up and use their local work directories. Is this correct behavior? Not
a big deal, but in order to clear the cluster state, I now need to
clear the work directory from each node, as well as, the gateway
data.

It's definitely debatable on what the correct behavior should be, but
was curious if this was the intent.

Thanks,
Paul

2

Hello Paul,

Am 15.09.2010 01:02 schrieb Paul:

Still in dev mode, so am clearing my data semi-frequently.

I noticed that on 0.10.0, when I clear the gateway, nodes still come
up and use their local work directories. Is this correct behavior? Not

it is, as long as not all data_nodes were shutdown. The gateway is just
for "longtime persistency", needed in case of a total breakdown of the
cluster. As long as at least one data_node has the index mirrored
locally (that is, it has its "running" index), all other nodes that may
join will get the data from that node, shared through the network-layer.
If all data_nodes are down and if you then bring one up newly, even if
there is a "local" index (Shay speaks of it as a temporary directory
), that index will be ignored and it is tried to load the shared gateway
(if we are speaking of the usage of a shared_filesystem, that is).

a big deal, but in order to clear the cluster state, I now need to
clear the work directory from each node, as well as, the gateway
data.

need not. Just shutdown the cluster e.g. with:
$ curl -XPOST 'http://localhost:9200/_cluster/nodes/_all/_shutdown'

It's definitely debatable on what the correct behavior should be, but
was curious if this was the intent.

it depends on the character of the gateway. The new "local"[1] gateway
will act differently I am sure but not experienced yet.
I am using ES version 0.9, but the behaviour should be the same on 0.10.

oo

[1]Gateway: a `local` gateway · Issue #343 · elastic/elasticsearch · GitHub

3

Hi Pascal,
Should have been more clear on my steps.

  1. Shutdown cluster
  2. Clear gateway data
  3. Start a node - I'd expect the node to start up empty, but instead
    loads indices from work directory, even though, gateway is empty.

Thanks

On Sep 15, 1:31 am, Pascal Christoph christ...@hbz-nrw.de wrote:

Hello Paul,

Am 15.09.2010 01:02 schrieb Paul:

Still in dev mode, so am clearing my data semi-frequently.

I noticed that on 0.10.0, when I clear the gateway, nodes still come
up and use their local work directories. Is this correct behavior? Not

it is, as long as not all data_nodes were shutdown. The gateway is just
for "longtime persistency", needed in case of a total breakdown of the
cluster. As long as at least one data_node has the index mirrored
locally (that is, it has its "running" index), all other nodes that may
join will get the data from that node, shared through the network-layer.
If all data_nodes are down and if you then bring one up newly, even if
there is a "local" index (Shay speaks of it as a temporary directory
), that index will be ignored and it is tried to load the shared gateway
(if we are speaking of the usage of a shared_filesystem, that is).

a big deal, but in order to clear the cluster state, I now need to
clear the work directory from each node, as well as, the gateway
data.

need not. Just shutdown the cluster e.g. with:
$ curl -XPOST 'http://localhost:9200/_cluster/nodes/_all/_shutdown'

It's definitely debatable on what the correct behavior should be, but
was curious if this was the intent.

it depends on the character of the gateway. The new "local"[1] gateway
will act differently I am sure but not experienced yet.
I am using ES version 0.9, but the behaviour should be the same on 0.10.

oo

[1]Gateway: a `local` gateway · Issue #343 · elastic/elasticsearch · GitHub

4

Hi Paul

Should have been more clear on my steps.

  1. Shutdown cluster
  2. Clear gateway data
  3. Start a node - I'd expect the node to start up empty, but instead
    loads indices from work directory, even though, gateway is empty.

Are you using 0.10.0 or master?

clint

5

I think Paul said 0.10 in the first email. Paul, when you say clearning the
gateway data, what exactly do you clean, I mean which directories? Do you
also clear the metadata? If so, then no indices will be created since there
is no metadata to reflect it that they ever existed in the gateway.

On Wed, Sep 15, 2010 at 5:38 PM, Clinton Gormley clinton@iannounce.co.ukwrote:

Hi Paul

Should have been more clear on my steps.

  1. Shutdown cluster
  2. Clear gateway data
  3. Start a node - I'd expect the node to start up empty, but instead
    loads indices from work directory, even though, gateway is empty.

Are you using 0.10.0 or master?

clint

6

In order to clear I am doing a rm -rf at the root of the gateway
directory, which deletes the indices and metaindex directories.

I tried to recreate a minimal test w/ 1 index w/ 1 doc to reproduce
and was not able to. This implies to me that something in my config or
setup is off. I am using the Node client from an external app, so
maybe that isn't shutting down and metadata is retained or something
like that.

Anyway, I need to dig more to see what is up.

Thanks for the replies.

On Sep 15, 12:53 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

I think Paul said 0.10 in the first email. Paul, when you say clearning the
gateway data, what exactly do you clean, I mean which directories? Do you
also clear the metadata? If so, then no indices will be created since there
is no metadata to reflect it that they ever existed in the gateway.

On Wed, Sep 15, 2010 at 5:38 PM, Clinton Gormley clin...@iannounce.co.ukwrote:

Hi Paul

Should have been more clear on my steps.

  1. Shutdown cluster
  2. Clear gateway data
  3. Start a node - I'd expect the node to start up empty, but instead
    loads indices from work directory, even though, gateway is empty.

Are you using 0.10.0 or master?

clint