Lost of index upon cluster restart


(Colin Surprenant) #1

Hi,

Yesterday I experienced a strange situation on my ES cluster; I
basically lost the last 10 days worth of indexed data after modifying
my config and resetting the cluster. I am not sure exactly how this
happened, and I haven't seen any particular problem in the logs. I'll
describe my setup and what I did and maybe someone :slight_smile: can help shed
some light on what I probably did wrong...

I have a 3 nodes cluster:

node1 is configured as:

  • http.enabled : true
  • node.data : false
  • index.gateway.type : fs
  • gateway.type : fs

node2 & node3:

  • http.enabled : false
  • node.data : true
  • index.gateway.type : fs
  • gateway.type : fs

I am using the default index storage settings (which should be niofs
since I am running on Linux) and the default shards and replicas
settings (5 shards and 1 replica).

Basically I changed the config file on my 3 nodes to modify my custom
stopwords configuration. (and btw, do I need to maintain this
stopwords config on both my data nodes and my http node??). I went and
did a "bin/service/elasticsearch restart" on my 3 nodes, one after the
other.

When the cluster was back up, there was a "hole" for the last 10 days
of indexed data. The logs do not show any particular problem.

What did I do wrong?

Thanks,
Colin


(Shay Banon) #2

I don't see where you define the location of the shared FS drive across
those 3 machines... . It is explained here:
http://www.elasticsearch.com/docs/elasticsearch/index_modules/gateway/fs/.
By default, just so elasticsearch can be simply tested with multi nodes on
the same machine, it goes to the work directory, but its local and not
shared... .

-shay.banon

On Thu, Jun 17, 2010 at 6:51 PM, Colin Surprenant <
colin.surprenant@gmail.com> wrote:

Hi,

Yesterday I experienced a strange situation on my ES cluster; I
basically lost the last 10 days worth of indexed data after modifying
my config and resetting the cluster. I am not sure exactly how this
happened, and I haven't seen any particular problem in the logs. I'll
describe my setup and what I did and maybe someone :slight_smile: can help shed
some light on what I probably did wrong...

I have a 3 nodes cluster:

node1 is configured as:

  • http.enabled : true
  • node.data : false
  • index.gateway.type : fs
  • gateway.type : fs

node2 & node3:

  • http.enabled : false
  • node.data : true
  • index.gateway.type : fs
  • gateway.type : fs

I am using the default index storage settings (which should be niofs
since I am running on Linux) and the default shards and replicas
settings (5 shards and 1 replica).

Basically I changed the config file on my 3 nodes to modify my custom
stopwords configuration. (and btw, do I need to maintain this
stopwords config on both my data nodes and my http node??). I went and
did a "bin/service/elasticsearch restart" on my 3 nodes, one after the
other.

When the cluster was back up, there was a "hole" for the last 10 days
of indexed data. The logs do not show any particular problem.

What did I do wrong?

Thanks,
Colin


(Colin Surprenant) #3

Ok, obviously, I did not get that and when I tested my config it was
multiple nodes on a single host...

I will then create a shared fs across my nodes and restart my cluster
to pickup the new config. Is there a way for me to restart my cluster
without loosing my indexes, in the state it is in now? (with a boggus
gateway config).

Also, in my config I have both:

  • index.gateway.type : fs
  • gateway.type : fs

Are both required?

All I need is to add this?

  • index.gateway.fs.location : /my/shared/path

Thanks for your help.
Colin

On Thu, Jun 17, 2010 at 2:51 PM, Shay Banon
shay.banon@elasticsearch.com wrote:

I don't see where you define the location of the shared FS drive across
those 3 machines... . It is explained
here: http://www.elasticsearch.com/docs/elasticsearch/index_modules/gateway/fs/.
By default, just so elasticsearch can be simply tested with multi nodes on
the same machine, it goes to the work directory, but its local and not
shared... .
-shay.banon

On Thu, Jun 17, 2010 at 6:51 PM, Colin Surprenant
colin.surprenant@gmail.com wrote:

Hi,

Yesterday I experienced a strange situation on my ES cluster; I
basically lost the last 10 days worth of indexed data after modifying
my config and resetting the cluster. I am not sure exactly how this
happened, and I haven't seen any particular problem in the logs. I'll
describe my setup and what I did and maybe someone :slight_smile: can help shed
some light on what I probably did wrong...

I have a 3 nodes cluster:

node1 is configured as:

  • http.enabled : true
  • node.data : false
  • index.gateway.type : fs
  • gateway.type : fs

node2 & node3:

  • http.enabled : false
  • node.data : true
  • index.gateway.type : fs
  • gateway.type : fs

I am using the default index storage settings (which should be niofs
since I am running on Linux) and the default shards and replicas
settings (5 shards and 1 replica).

Basically I changed the config file on my 3 nodes to modify my custom
stopwords configuration. (and btw, do I need to maintain this
stopwords config on both my data nodes and my http node??). I went and
did a "bin/service/elasticsearch restart" on my 3 nodes, one after the
other.

When the cluster was back up, there was a "hole" for the last 10 days
of indexed data. The logs do not show any particular problem.

What did I do wrong?

Thanks,
Colin


(Colin Surprenant) #4

I will finally go with the S3 cloud gateway using config like:

gateway:
type: cloud
cloud:
chunk_size: 1g
container: mycontainer
index :
gateway :
type : cloud

Now, is the last "index.gateway.type : cloud" config needed or it is
completely redundant to the node level gateway config? What I
understand from the doc is that the index level gateway will inherit
from the node level gateway and index level gateway config is only
required to specify exceptions to the node level gateway config.

Thanks,
Colin

On Thu, Jun 17, 2010 at 3:25 PM, Colin Surprenant
colin.surprenant@gmail.com wrote:

Ok, obviously, I did not get that and when I tested my config it was
multiple nodes on a single host...

I will then create a shared fs across my nodes and restart my cluster
to pickup the new config. Is there a way for me to restart my cluster
without loosing my indexes, in the state it is in now? (with a boggus
gateway config).

Also, in my config I have both:

  • index.gateway.type : fs
  • gateway.type : fs

Are both required?

All I need is to add this?

  • index.gateway.fs.location : /my/shared/path

Thanks for your help.
Colin

On Thu, Jun 17, 2010 at 2:51 PM, Shay Banon
shay.banon@elasticsearch.com wrote:

I don't see where you define the location of the shared FS drive across
those 3 machines... . It is explained
here: http://www.elasticsearch.com/docs/elasticsearch/index_modules/gateway/fs/.
By default, just so elasticsearch can be simply tested with multi nodes on
the same machine, it goes to the work directory, but its local and not
shared... .
-shay.banon

On Thu, Jun 17, 2010 at 6:51 PM, Colin Surprenant
colin.surprenant@gmail.com wrote:

Hi,

Yesterday I experienced a strange situation on my ES cluster; I
basically lost the last 10 days worth of indexed data after modifying
my config and resetting the cluster. I am not sure exactly how this
happened, and I haven't seen any particular problem in the logs. I'll
describe my setup and what I did and maybe someone :slight_smile: can help shed
some light on what I probably did wrong...

I have a 3 nodes cluster:

node1 is configured as:

  • http.enabled : true
  • node.data : false
  • index.gateway.type : fs
  • gateway.type : fs

node2 & node3:

  • http.enabled : false
  • node.data : true
  • index.gateway.type : fs
  • gateway.type : fs

I am using the default index storage settings (which should be niofs
since I am running on Linux) and the default shards and replicas
settings (5 shards and 1 replica).

Basically I changed the config file on my 3 nodes to modify my custom
stopwords configuration. (and btw, do I need to maintain this
stopwords config on both my data nodes and my http node??). I went and
did a "bin/service/elasticsearch restart" on my 3 nodes, one after the
other.

When the cluster was back up, there was a "hole" for the last 10 days
of indexed data. The logs do not show any particular problem.

What did I do wrong?

Thanks,
Colin


(Shay Banon) #5

You don't need to set the index.gateway.type, since it will, as you pointed
out, inherit the gateway type. Regarding the first setting of shared fs
gateway, you need to set the location on the gateway level:

gateway:
type: fs
fs:
location: /mnt/sharedfs

You should not set the location of the index gateway (its there, but for
really advance use cases). The location on the gateway level is important
since that where the cluster meta data will be stored.

A disclaimer regarding the cloud gateway, i have hear people having problem
with it (caused by jclouds). For 0.9 I am going to write a native AWS cloud
version.

-shay.banon

On Thu, Jun 17, 2010 at 11:12 PM, Colin Surprenant <
colin.surprenant@gmail.com> wrote:

I will finally go with the S3 cloud gateway using config like:

gateway:
type: cloud
cloud:
chunk_size: 1g
container: mycontainer
index :
gateway :
type : cloud

Now, is the last "index.gateway.type : cloud" config needed or it is
completely redundant to the node level gateway config? What I
understand from the doc is that the index level gateway will inherit
from the node level gateway and index level gateway config is only
required to specify exceptions to the node level gateway config.

Thanks,
Colin

On Thu, Jun 17, 2010 at 3:25 PM, Colin Surprenant
colin.surprenant@gmail.com wrote:

Ok, obviously, I did not get that and when I tested my config it was
multiple nodes on a single host...

I will then create a shared fs across my nodes and restart my cluster
to pickup the new config. Is there a way for me to restart my cluster
without loosing my indexes, in the state it is in now? (with a boggus
gateway config).

Also, in my config I have both:

  • index.gateway.type : fs
  • gateway.type : fs

Are both required?

All I need is to add this?

  • index.gateway.fs.location : /my/shared/path

Thanks for your help.
Colin

On Thu, Jun 17, 2010 at 2:51 PM, Shay Banon
shay.banon@elasticsearch.com wrote:

I don't see where you define the location of the shared FS drive across
those 3 machines... . It is explained
here:
http://www.elasticsearch.com/docs/elasticsearch/index_modules/gateway/fs/.

By default, just so elasticsearch can be simply tested with multi nodes
on

the same machine, it goes to the work directory, but its local and not
shared... .
-shay.banon

On Thu, Jun 17, 2010 at 6:51 PM, Colin Surprenant
colin.surprenant@gmail.com wrote:

Hi,

Yesterday I experienced a strange situation on my ES cluster; I
basically lost the last 10 days worth of indexed data after modifying
my config and resetting the cluster. I am not sure exactly how this
happened, and I haven't seen any particular problem in the logs. I'll
describe my setup and what I did and maybe someone :slight_smile: can help shed
some light on what I probably did wrong...

I have a 3 nodes cluster:

node1 is configured as:

  • http.enabled : true
  • node.data : false
  • index.gateway.type : fs
  • gateway.type : fs

node2 & node3:

  • http.enabled : false
  • node.data : true
  • index.gateway.type : fs
  • gateway.type : fs

I am using the default index storage settings (which should be niofs
since I am running on Linux) and the default shards and replicas
settings (5 shards and 1 replica).

Basically I changed the config file on my 3 nodes to modify my custom
stopwords configuration. (and btw, do I need to maintain this
stopwords config on both my data nodes and my http node??). I went and
did a "bin/service/elasticsearch restart" on my 3 nodes, one after the
other.

When the cluster was back up, there was a "hole" for the last 10 days
of indexed data. The logs do not show any particular problem.

What did I do wrong?

Thanks,
Colin


(Colin Surprenant) #6

Thanks for your help Shay.

Ok, so to play it safe, I'll go with the shared fs gateway for now.

Regarding the config change + required cluster restart, since my
current cluster gateway config is boggus (using local fs storage for
gateway) what would be the best procedure to restart my cluster
hopefully without loosing my current indexes? Again, my setup is 1
non-data+http node and 2 data+non-http nodes.

Thanks,
Colin

On Thu, Jun 17, 2010 at 4:53 PM, Shay Banon
shay.banon@elasticsearch.com wrote:

You don't need to set the index.gateway.type, since it will, as you pointed
out, inherit the gateway type. Regarding the first setting of shared fs
gateway, you need to set the location on the gateway level:
gateway:
type: fs
fs:
location: /mnt/sharedfs
You should not set the location of the index gateway (its there, but for
really advance use cases). The location on the gateway level is important
since that where the cluster meta data will be stored.
A disclaimer regarding the cloud gateway, i have hear people having problem
with it (caused by jclouds). For 0.9 I am going to write a native AWS cloud
version.
-shay.banon

On Thu, Jun 17, 2010 at 11:12 PM, Colin Surprenant
colin.surprenant@gmail.com wrote:

I will finally go with the S3 cloud gateway using config like:

gateway:
type: cloud
cloud:
chunk_size: 1g
container: mycontainer
index :
gateway :
type : cloud

Now, is the last "index.gateway.type : cloud" config needed or it is
completely redundant to the node level gateway config? What I
understand from the doc is that the index level gateway will inherit
from the node level gateway and index level gateway config is only
required to specify exceptions to the node level gateway config.

Thanks,
Colin

On Thu, Jun 17, 2010 at 3:25 PM, Colin Surprenant
colin.surprenant@gmail.com wrote:

Ok, obviously, I did not get that and when I tested my config it was
multiple nodes on a single host...

I will then create a shared fs across my nodes and restart my cluster
to pickup the new config. Is there a way for me to restart my cluster
without loosing my indexes, in the state it is in now? (with a boggus
gateway config).

Also, in my config I have both:

  • index.gateway.type : fs
  • gateway.type : fs

Are both required?

All I need is to add this?

  • index.gateway.fs.location : /my/shared/path

Thanks for your help.
Colin

On Thu, Jun 17, 2010 at 2:51 PM, Shay Banon
shay.banon@elasticsearch.com wrote:

I don't see where you define the location of the shared FS drive across
those 3 machines... . It is explained

here: http://www.elasticsearch.com/docs/elasticsearch/index_modules/gateway/fs/.
By default, just so elasticsearch can be simply tested with multi nodes
on
the same machine, it goes to the work directory, but its local and not
shared... .
-shay.banon

On Thu, Jun 17, 2010 at 6:51 PM, Colin Surprenant
colin.surprenant@gmail.com wrote:

Hi,

Yesterday I experienced a strange situation on my ES cluster; I
basically lost the last 10 days worth of indexed data after modifying
my config and resetting the cluster. I am not sure exactly how this
happened, and I haven't seen any particular problem in the logs. I'll
describe my setup and what I did and maybe someone :slight_smile: can help shed
some light on what I probably did wrong...

I have a 3 nodes cluster:

node1 is configured as:

  • http.enabled : true
  • node.data : false
  • index.gateway.type : fs
  • gateway.type : fs

node2 & node3:

  • http.enabled : false
  • node.data : true
  • index.gateway.type : fs
  • gateway.type : fs

I am using the default index storage settings (which should be niofs
since I am running on Linux) and the default shards and replicas
settings (5 shards and 1 replica).

Basically I changed the config file on my 3 nodes to modify my custom
stopwords configuration. (and btw, do I need to maintain this
stopwords config on both my data nodes and my http node??). I went and
did a "bin/service/elasticsearch restart" on my 3 nodes, one after the
other.

When the cluster was back up, there was a "hole" for the last 10 days
of indexed data. The logs do not show any particular problem.

What did I do wrong?

Thanks,
Colin


(Shay Banon) #7

Can you share your latest configuration, I got several on this thread...

-shay.banon

On Fri, Jun 18, 2010 at 12:03 AM, Colin Surprenant <
colin.surprenant@gmail.com> wrote:

Thanks for your help Shay.

Ok, so to play it safe, I'll go with the shared fs gateway for now.

Regarding the config change + required cluster restart, since my
current cluster gateway config is boggus (using local fs storage for
gateway) what would be the best procedure to restart my cluster
hopefully without loosing my current indexes? Again, my setup is 1
non-data+http node and 2 data+non-http nodes.

Thanks,
Colin

On Thu, Jun 17, 2010 at 4:53 PM, Shay Banon
shay.banon@elasticsearch.com wrote:

You don't need to set the index.gateway.type, since it will, as you
pointed
out, inherit the gateway type. Regarding the first setting of shared fs
gateway, you need to set the location on the gateway level:
gateway:
type: fs
fs:
location: /mnt/sharedfs
You should not set the location of the index gateway (its there, but for
really advance use cases). The location on the gateway level is important
since that where the cluster meta data will be stored.
A disclaimer regarding the cloud gateway, i have hear people having
problem
with it (caused by jclouds). For 0.9 I am going to write a native AWS
cloud
version.
-shay.banon

On Thu, Jun 17, 2010 at 11:12 PM, Colin Surprenant
colin.surprenant@gmail.com wrote:

I will finally go with the S3 cloud gateway using config like:

gateway:
type: cloud
cloud:
chunk_size: 1g
container: mycontainer
index :
gateway :
type : cloud

Now, is the last "index.gateway.type : cloud" config needed or it is
completely redundant to the node level gateway config? What I
understand from the doc is that the index level gateway will inherit
from the node level gateway and index level gateway config is only
required to specify exceptions to the node level gateway config.

Thanks,
Colin

On Thu, Jun 17, 2010 at 3:25 PM, Colin Surprenant
colin.surprenant@gmail.com wrote:

Ok, obviously, I did not get that and when I tested my config it was
multiple nodes on a single host...

I will then create a shared fs across my nodes and restart my cluster
to pickup the new config. Is there a way for me to restart my cluster
without loosing my indexes, in the state it is in now? (with a boggus
gateway config).

Also, in my config I have both:

  • index.gateway.type : fs
  • gateway.type : fs

Are both required?

All I need is to add this?

  • index.gateway.fs.location : /my/shared/path

Thanks for your help.
Colin

On Thu, Jun 17, 2010 at 2:51 PM, Shay Banon
shay.banon@elasticsearch.com wrote:

I don't see where you define the location of the shared FS drive
across

those 3 machines... . It is explained

here:
http://www.elasticsearch.com/docs/elasticsearch/index_modules/gateway/fs/.

By default, just so elasticsearch can be simply tested with multi
nodes

on
the same machine, it goes to the work directory, but its local and
not

shared... .
-shay.banon

On Thu, Jun 17, 2010 at 6:51 PM, Colin Surprenant
colin.surprenant@gmail.com wrote:

Hi,

Yesterday I experienced a strange situation on my ES cluster; I
basically lost the last 10 days worth of indexed data after
modifying

my config and resetting the cluster. I am not sure exactly how this
happened, and I haven't seen any particular problem in the logs.
I'll

describe my setup and what I did and maybe someone :slight_smile: can help shed
some light on what I probably did wrong...

I have a 3 nodes cluster:

node1 is configured as:

  • http.enabled : true
  • node.data : false
  • index.gateway.type : fs
  • gateway.type : fs

node2 & node3:

  • http.enabled : false
  • node.data : true
  • index.gateway.type : fs
  • gateway.type : fs

I am using the default index storage settings (which should be niofs
since I am running on Linux) and the default shards and replicas
settings (5 shards and 1 replica).

Basically I changed the config file on my 3 nodes to modify my
custom

stopwords configuration. (and btw, do I need to maintain this
stopwords config on both my data nodes and my http node??). I went
and

did a "bin/service/elasticsearch restart" on my 3 nodes, one after
the

other.

When the cluster was back up, there was a "hole" for the last 10
days

of indexed data. The logs do not show any particular problem.

What did I do wrong?

Thanks,
Colin


(Colin Surprenant) #8

I have a 3 nodes clusters, with one http+nondata node and two
nonhttp+data nodes.

  • node 1 config:

http :
enabled : true
node :
data : false
gateway :
type : fs

  • node 2 & 3

http :
enabled : false
node :
data : true
gateway :
type : fs

So the question is, since this current setup is boggus because the
gateway fs path is not shared between the nodes, what is my best
course of action to correct the problem without loosing my indexes.
Since the data sits on nodes 2 & 3, if I restart node 2 with the new
config, wait for it to get the replication back from node 3 then
proceed with the restart on node 3, wait for replication from node 2
and then finally restart node 1, would that work?

I will either use NFS (I'd rather not) or HDFS (see other thread
"cloud gateway") for the gateway.

Thanks,
Colin

On Thu, Jun 17, 2010 at 5:41 PM, Shay Banon
shay.banon@elasticsearch.com wrote:

Can you share your latest configuration, I got several on this thread...
-shay.banon

On Fri, Jun 18, 2010 at 12:03 AM, Colin Surprenant
colin.surprenant@gmail.com wrote:

Thanks for your help Shay.

Ok, so to play it safe, I'll go with the shared fs gateway for now.

Regarding the config change + required cluster restart, since my
current cluster gateway config is boggus (using local fs storage for
gateway) what would be the best procedure to restart my cluster
hopefully without loosing my current indexes? Again, my setup is 1
non-data+http node and 2 data+non-http nodes.

Thanks,
Colin

On Thu, Jun 17, 2010 at 4:53 PM, Shay Banon
shay.banon@elasticsearch.com wrote:

You don't need to set the index.gateway.type, since it will, as you
pointed
out, inherit the gateway type. Regarding the first setting of shared fs
gateway, you need to set the location on the gateway level:
gateway:
type: fs
fs:
location: /mnt/sharedfs
You should not set the location of the index gateway (its there, but for
really advance use cases). The location on the gateway level is
important
since that where the cluster meta data will be stored.
A disclaimer regarding the cloud gateway, i have hear people having
problem
with it (caused by jclouds). For 0.9 I am going to write a native AWS
cloud
version.
-shay.banon

On Thu, Jun 17, 2010 at 11:12 PM, Colin Surprenant
colin.surprenant@gmail.com wrote:

I will finally go with the S3 cloud gateway using config like:

gateway:
type: cloud
cloud:
chunk_size: 1g
container: mycontainer
index :
gateway :
type : cloud

Now, is the last "index.gateway.type : cloud" config needed or it is
completely redundant to the node level gateway config? What I
understand from the doc is that the index level gateway will inherit
from the node level gateway and index level gateway config is only
required to specify exceptions to the node level gateway config.

Thanks,
Colin

On Thu, Jun 17, 2010 at 3:25 PM, Colin Surprenant
colin.surprenant@gmail.com wrote:

Ok, obviously, I did not get that and when I tested my config it was
multiple nodes on a single host...

I will then create a shared fs across my nodes and restart my cluster
to pickup the new config. Is there a way for me to restart my cluster
without loosing my indexes, in the state it is in now? (with a boggus
gateway config).

Also, in my config I have both:

  • index.gateway.type : fs
  • gateway.type : fs

Are both required?

All I need is to add this?

  • index.gateway.fs.location : /my/shared/path

Thanks for your help.
Colin

On Thu, Jun 17, 2010 at 2:51 PM, Shay Banon
shay.banon@elasticsearch.com wrote:

I don't see where you define the location of the shared FS drive
across
those 3 machines... . It is explained

here: http://www.elasticsearch.com/docs/elasticsearch/index_modules/gateway/fs/.
By default, just so elasticsearch can be simply tested with multi
nodes
on
the same machine, it goes to the work directory, but its local and
not
shared... .
-shay.banon

On Thu, Jun 17, 2010 at 6:51 PM, Colin Surprenant
colin.surprenant@gmail.com wrote:

Hi,

Yesterday I experienced a strange situation on my ES cluster; I
basically lost the last 10 days worth of indexed data after
modifying
my config and resetting the cluster. I am not sure exactly how this
happened, and I haven't seen any particular problem in the logs.
I'll
describe my setup and what I did and maybe someone :slight_smile: can help shed
some light on what I probably did wrong...

I have a 3 nodes cluster:

node1 is configured as:

  • http.enabled : true
  • node.data : false
  • index.gateway.type : fs
  • gateway.type : fs

node2 & node3:

  • http.enabled : false
  • node.data : true
  • index.gateway.type : fs
  • gateway.type : fs

I am using the default index storage settings (which should be
niofs
since I am running on Linux) and the default shards and replicas
settings (5 shards and 1 replica).

Basically I changed the config file on my 3 nodes to modify my
custom
stopwords configuration. (and btw, do I need to maintain this
stopwords config on both my data nodes and my http node??). I went
and
did a "bin/service/elasticsearch restart" on my 3 nodes, one after
the
other.

When the cluster was back up, there was a "hole" for the last 10
days
of indexed data. The logs do not show any particular problem.

What did I do wrong?

Thanks,
Colin


(Shay Banon) #9

If you are going to use FS / HDFS, you can use the cluster state API to find
out which node is the primary, then copy over the content of the FS gateway
(under the work directory) and copy it over.

-shay.banon

On Fri, Jun 18, 2010 at 5:22 PM, Colin Surprenant <
colin.surprenant@gmail.com> wrote:

I have a 3 nodes clusters, with one http+nondata node and two
nonhttp+data nodes.

  • node 1 config:

http :
enabled : true
node :
data : false
gateway :
type : fs

  • node 2 & 3

http :
enabled : false
node :
data : true
gateway :
type : fs

So the question is, since this current setup is boggus because the
gateway fs path is not shared between the nodes, what is my best
course of action to correct the problem without loosing my indexes.
Since the data sits on nodes 2 & 3, if I restart node 2 with the new
config, wait for it to get the replication back from node 3 then
proceed with the restart on node 3, wait for replication from node 2
and then finally restart node 1, would that work?

I will either use NFS (I'd rather not) or HDFS (see other thread
"cloud gateway") for the gateway.

Thanks,
Colin

On Thu, Jun 17, 2010 at 5:41 PM, Shay Banon
shay.banon@elasticsearch.com wrote:

Can you share your latest configuration, I got several on this thread...
-shay.banon

On Fri, Jun 18, 2010 at 12:03 AM, Colin Surprenant
colin.surprenant@gmail.com wrote:

Thanks for your help Shay.

Ok, so to play it safe, I'll go with the shared fs gateway for now.

Regarding the config change + required cluster restart, since my
current cluster gateway config is boggus (using local fs storage for
gateway) what would be the best procedure to restart my cluster
hopefully without loosing my current indexes? Again, my setup is 1
non-data+http node and 2 data+non-http nodes.

Thanks,
Colin

On Thu, Jun 17, 2010 at 4:53 PM, Shay Banon
shay.banon@elasticsearch.com wrote:

You don't need to set the index.gateway.type, since it will, as you
pointed
out, inherit the gateway type. Regarding the first setting of shared
fs

gateway, you need to set the location on the gateway level:
gateway:
type: fs
fs:
location: /mnt/sharedfs
You should not set the location of the index gateway (its there, but
for

really advance use cases). The location on the gateway level is
important
since that where the cluster meta data will be stored.
A disclaimer regarding the cloud gateway, i have hear people having
problem
with it (caused by jclouds). For 0.9 I am going to write a native AWS
cloud
version.
-shay.banon

On Thu, Jun 17, 2010 at 11:12 PM, Colin Surprenant
colin.surprenant@gmail.com wrote:

I will finally go with the S3 cloud gateway using config like:

gateway:
type: cloud
cloud:
chunk_size: 1g
container: mycontainer
index :
gateway :
type : cloud

Now, is the last "index.gateway.type : cloud" config needed or it is
completely redundant to the node level gateway config? What I
understand from the doc is that the index level gateway will inherit
from the node level gateway and index level gateway config is only
required to specify exceptions to the node level gateway config.

Thanks,
Colin

On Thu, Jun 17, 2010 at 3:25 PM, Colin Surprenant
colin.surprenant@gmail.com wrote:

Ok, obviously, I did not get that and when I tested my config it
was

multiple nodes on a single host...

I will then create a shared fs across my nodes and restart my
cluster

to pickup the new config. Is there a way for me to restart my
cluster

without loosing my indexes, in the state it is in now? (with a
boggus

gateway config).

Also, in my config I have both:

  • index.gateway.type : fs
  • gateway.type : fs

Are both required?

All I need is to add this?

  • index.gateway.fs.location : /my/shared/path

Thanks for your help.
Colin

On Thu, Jun 17, 2010 at 2:51 PM, Shay Banon
shay.banon@elasticsearch.com wrote:

I don't see where you define the location of the shared FS drive
across
those 3 machines... . It is explained

here:
http://www.elasticsearch.com/docs/elasticsearch/index_modules/gateway/fs/.

By default, just so elasticsearch can be simply tested with multi
nodes
on
the same machine, it goes to the work directory, but its local and
not
shared... .
-shay.banon

On Thu, Jun 17, 2010 at 6:51 PM, Colin Surprenant
colin.surprenant@gmail.com wrote:

Hi,

Yesterday I experienced a strange situation on my ES cluster; I
basically lost the last 10 days worth of indexed data after
modifying
my config and resetting the cluster. I am not sure exactly how
this

happened, and I haven't seen any particular problem in the logs.
I'll
describe my setup and what I did and maybe someone :slight_smile: can help
shed

some light on what I probably did wrong...

I have a 3 nodes cluster:

node1 is configured as:

  • http.enabled : true
  • node.data : false
  • index.gateway.type : fs
  • gateway.type : fs

node2 & node3:

  • http.enabled : false
  • node.data : true
  • index.gateway.type : fs
  • gateway.type : fs

I am using the default index storage settings (which should be
niofs
since I am running on Linux) and the default shards and replicas
settings (5 shards and 1 replica).

Basically I changed the config file on my 3 nodes to modify my
custom
stopwords configuration. (and btw, do I need to maintain this
stopwords config on both my data nodes and my http node??). I
went

and
did a "bin/service/elasticsearch restart" on my 3 nodes, one
after

the
other.

When the cluster was back up, there was a "hole" for the last 10
days
of indexed data. The logs do not show any particular problem.

What did I do wrong?

Thanks,
Colin


(system) #10