Yesterday I experienced a strange situation on my ES cluster; I
basically lost the last 10 days worth of indexed data after modifying
my config and resetting the cluster. I am not sure exactly how this
happened, and I haven't seen any particular problem in the logs. I'll
describe my setup and what I did and maybe someone can help shed
some light on what I probably did wrong...
I have a 3 nodes cluster:
node1 is configured as:
http.enabled : true
node.data : false
index.gateway.type : fs
gateway.type : fs
node2 & node3:
http.enabled : false
node.data : true
index.gateway.type : fs
gateway.type : fs
I am using the default index storage settings (which should be niofs
since I am running on Linux) and the default shards and replicas
settings (5 shards and 1 replica).
Basically I changed the config file on my 3 nodes to modify my custom
stopwords configuration. (and btw, do I need to maintain this
stopwords config on both my data nodes and my http node??). I went and
did a "bin/service/elasticsearch restart" on my 3 nodes, one after the
other.
When the cluster was back up, there was a "hole" for the last 10 days
of indexed data. The logs do not show any particular problem.
I don't see where you define the location of the shared FS drive across
those 3 machines... . It is explained here: http://www.elasticsearch.com/docs/elasticsearch/index_modules/gateway/fs/.
By default, just so elasticsearch can be simply tested with multi nodes on
the same machine, it goes to the work directory, but its local and not
shared... .
Yesterday I experienced a strange situation on my ES cluster; I
basically lost the last 10 days worth of indexed data after modifying
my config and resetting the cluster. I am not sure exactly how this
happened, and I haven't seen any particular problem in the logs. I'll
describe my setup and what I did and maybe someone can help shed
some light on what I probably did wrong...
I have a 3 nodes cluster:
node1 is configured as:
http.enabled : true
node.data : false
index.gateway.type : fs
gateway.type : fs
node2 & node3:
http.enabled : false
node.data : true
index.gateway.type : fs
gateway.type : fs
I am using the default index storage settings (which should be niofs
since I am running on Linux) and the default shards and replicas
settings (5 shards and 1 replica).
Basically I changed the config file on my 3 nodes to modify my custom
stopwords configuration. (and btw, do I need to maintain this
stopwords config on both my data nodes and my http node??). I went and
did a "bin/service/elasticsearch restart" on my 3 nodes, one after the
other.
When the cluster was back up, there was a "hole" for the last 10 days
of indexed data. The logs do not show any particular problem.
Ok, obviously, I did not get that and when I tested my config it was
multiple nodes on a single host...
I will then create a shared fs across my nodes and restart my cluster
to pickup the new config. Is there a way for me to restart my cluster
without loosing my indexes, in the state it is in now? (with a boggus
gateway config).
I don't see where you define the location of the shared FS drive across
those 3 machines... . It is explained
here: http://www.elasticsearch.com/docs/elasticsearch/index_modules/gateway/fs/.
By default, just so elasticsearch can be simply tested with multi nodes on
the same machine, it goes to the work directory, but its local and not
shared... .
-shay.banon
Yesterday I experienced a strange situation on my ES cluster; I
basically lost the last 10 days worth of indexed data after modifying
my config and resetting the cluster. I am not sure exactly how this
happened, and I haven't seen any particular problem in the logs. I'll
describe my setup and what I did and maybe someone can help shed
some light on what I probably did wrong...
I have a 3 nodes cluster:
node1 is configured as:
http.enabled : true
node.data : false
index.gateway.type : fs
gateway.type : fs
node2 & node3:
http.enabled : false
node.data : true
index.gateway.type : fs
gateway.type : fs
I am using the default index storage settings (which should be niofs
since I am running on Linux) and the default shards and replicas
settings (5 shards and 1 replica).
Basically I changed the config file on my 3 nodes to modify my custom
stopwords configuration. (and btw, do I need to maintain this
stopwords config on both my data nodes and my http node??). I went and
did a "bin/service/elasticsearch restart" on my 3 nodes, one after the
other.
When the cluster was back up, there was a "hole" for the last 10 days
of indexed data. The logs do not show any particular problem.
I will finally go with the S3 cloud gateway using config like:
gateway:
type: cloud
cloud:
chunk_size: 1g
container: mycontainer
index :
gateway :
type : cloud
Now, is the last "index.gateway.type : cloud" config needed or it is
completely redundant to the node level gateway config? What I
understand from the doc is that the index level gateway will inherit
from the node level gateway and index level gateway config is only
required to specify exceptions to the node level gateway config.
Ok, obviously, I did not get that and when I tested my config it was
multiple nodes on a single host...
I will then create a shared fs across my nodes and restart my cluster
to pickup the new config. Is there a way for me to restart my cluster
without loosing my indexes, in the state it is in now? (with a boggus
gateway config).
I don't see where you define the location of the shared FS drive across
those 3 machines... . It is explained
here: http://www.elasticsearch.com/docs/elasticsearch/index_modules/gateway/fs/.
By default, just so elasticsearch can be simply tested with multi nodes on
the same machine, it goes to the work directory, but its local and not
shared... .
-shay.banon
Yesterday I experienced a strange situation on my ES cluster; I
basically lost the last 10 days worth of indexed data after modifying
my config and resetting the cluster. I am not sure exactly how this
happened, and I haven't seen any particular problem in the logs. I'll
describe my setup and what I did and maybe someone can help shed
some light on what I probably did wrong...
I have a 3 nodes cluster:
node1 is configured as:
http.enabled : true
node.data : false
index.gateway.type : fs
gateway.type : fs
node2 & node3:
http.enabled : false
node.data : true
index.gateway.type : fs
gateway.type : fs
I am using the default index storage settings (which should be niofs
since I am running on Linux) and the default shards and replicas
settings (5 shards and 1 replica).
Basically I changed the config file on my 3 nodes to modify my custom
stopwords configuration. (and btw, do I need to maintain this
stopwords config on both my data nodes and my http node??). I went and
did a "bin/service/elasticsearch restart" on my 3 nodes, one after the
other.
When the cluster was back up, there was a "hole" for the last 10 days
of indexed data. The logs do not show any particular problem.
You don't need to set the index.gateway.type, since it will, as you pointed
out, inherit the gateway type. Regarding the first setting of shared fs
gateway, you need to set the location on the gateway level:
gateway:
type: fs
fs:
location: /mnt/sharedfs
You should not set the location of the index gateway (its there, but for
really advance use cases). The location on the gateway level is important
since that where the cluster meta data will be stored.
A disclaimer regarding the cloud gateway, i have hear people having problem
with it (caused by jclouds). For 0.9 I am going to write a native AWS cloud
version.
I will finally go with the S3 cloud gateway using config like:
gateway:
type: cloud
cloud:
chunk_size: 1g
container: mycontainer
index :
gateway :
type : cloud
Now, is the last "index.gateway.type : cloud" config needed or it is
completely redundant to the node level gateway config? What I
understand from the doc is that the index level gateway will inherit
from the node level gateway and index level gateway config is only
required to specify exceptions to the node level gateway config.
Ok, obviously, I did not get that and when I tested my config it was
multiple nodes on a single host...
I will then create a shared fs across my nodes and restart my cluster
to pickup the new config. Is there a way for me to restart my cluster
without loosing my indexes, in the state it is in now? (with a boggus
gateway config).
I don't see where you define the location of the shared FS drive across
those 3 machines... . It is explained
here: http://www.elasticsearch.com/docs/elasticsearch/index_modules/gateway/fs/.
By default, just so elasticsearch can be simply tested with multi nodes
on
the same machine, it goes to the work directory, but its local and not
shared... .
-shay.banon
Yesterday I experienced a strange situation on my ES cluster; I
basically lost the last 10 days worth of indexed data after modifying
my config and resetting the cluster. I am not sure exactly how this
happened, and I haven't seen any particular problem in the logs. I'll
describe my setup and what I did and maybe someone can help shed
some light on what I probably did wrong...
I have a 3 nodes cluster:
node1 is configured as:
http.enabled : true
node.data : false
index.gateway.type : fs
gateway.type : fs
node2 & node3:
http.enabled : false
node.data : true
index.gateway.type : fs
gateway.type : fs
I am using the default index storage settings (which should be niofs
since I am running on Linux) and the default shards and replicas
settings (5 shards and 1 replica).
Basically I changed the config file on my 3 nodes to modify my custom
stopwords configuration. (and btw, do I need to maintain this
stopwords config on both my data nodes and my http node??). I went and
did a "bin/service/elasticsearch restart" on my 3 nodes, one after the
other.
When the cluster was back up, there was a "hole" for the last 10 days
of indexed data. The logs do not show any particular problem.
Ok, so to play it safe, I'll go with the shared fs gateway for now.
Regarding the config change + required cluster restart, since my
current cluster gateway config is boggus (using local fs storage for
gateway) what would be the best procedure to restart my cluster
hopefully without loosing my current indexes? Again, my setup is 1
non-data+http node and 2 data+non-http nodes.
You don't need to set the index.gateway.type, since it will, as you pointed
out, inherit the gateway type. Regarding the first setting of shared fs
gateway, you need to set the location on the gateway level:
gateway:
type: fs
fs:
location: /mnt/sharedfs
You should not set the location of the index gateway (its there, but for
really advance use cases). The location on the gateway level is important
since that where the cluster meta data will be stored.
A disclaimer regarding the cloud gateway, i have hear people having problem
with it (caused by jclouds). For 0.9 I am going to write a native AWS cloud
version.
-shay.banon
I will finally go with the S3 cloud gateway using config like:
gateway:
type: cloud
cloud:
chunk_size: 1g
container: mycontainer
index :
gateway :
type : cloud
Now, is the last "index.gateway.type : cloud" config needed or it is
completely redundant to the node level gateway config? What I
understand from the doc is that the index level gateway will inherit
from the node level gateway and index level gateway config is only
required to specify exceptions to the node level gateway config.
Ok, obviously, I did not get that and when I tested my config it was
multiple nodes on a single host...
I will then create a shared fs across my nodes and restart my cluster
to pickup the new config. Is there a way for me to restart my cluster
without loosing my indexes, in the state it is in now? (with a boggus
gateway config).
Yesterday I experienced a strange situation on my ES cluster; I
basically lost the last 10 days worth of indexed data after modifying
my config and resetting the cluster. I am not sure exactly how this
happened, and I haven't seen any particular problem in the logs. I'll
describe my setup and what I did and maybe someone can help shed
some light on what I probably did wrong...
I have a 3 nodes cluster:
node1 is configured as:
http.enabled : true
node.data : false
index.gateway.type : fs
gateway.type : fs
node2 & node3:
http.enabled : false
node.data : true
index.gateway.type : fs
gateway.type : fs
I am using the default index storage settings (which should be niofs
since I am running on Linux) and the default shards and replicas
settings (5 shards and 1 replica).
Basically I changed the config file on my 3 nodes to modify my custom
stopwords configuration. (and btw, do I need to maintain this
stopwords config on both my data nodes and my http node??). I went and
did a "bin/service/elasticsearch restart" on my 3 nodes, one after the
other.
When the cluster was back up, there was a "hole" for the last 10 days
of indexed data. The logs do not show any particular problem.
Ok, so to play it safe, I'll go with the shared fs gateway for now.
Regarding the config change + required cluster restart, since my
current cluster gateway config is boggus (using local fs storage for
gateway) what would be the best procedure to restart my cluster
hopefully without loosing my current indexes? Again, my setup is 1
non-data+http node and 2 data+non-http nodes.
You don't need to set the index.gateway.type, since it will, as you
pointed
out, inherit the gateway type. Regarding the first setting of shared fs
gateway, you need to set the location on the gateway level:
gateway:
type: fs
fs:
location: /mnt/sharedfs
You should not set the location of the index gateway (its there, but for
really advance use cases). The location on the gateway level is important
since that where the cluster meta data will be stored.
A disclaimer regarding the cloud gateway, i have hear people having
problem
with it (caused by jclouds). For 0.9 I am going to write a native AWS
cloud
version.
-shay.banon
I will finally go with the S3 cloud gateway using config like:
gateway:
type: cloud
cloud:
chunk_size: 1g
container: mycontainer
index :
gateway :
type : cloud
Now, is the last "index.gateway.type : cloud" config needed or it is
completely redundant to the node level gateway config? What I
understand from the doc is that the index level gateway will inherit
from the node level gateway and index level gateway config is only
required to specify exceptions to the node level gateway config.
Ok, obviously, I did not get that and when I tested my config it was
multiple nodes on a single host...
I will then create a shared fs across my nodes and restart my cluster
to pickup the new config. Is there a way for me to restart my cluster
without loosing my indexes, in the state it is in now? (with a boggus
gateway config).
Yesterday I experienced a strange situation on my ES cluster; I
basically lost the last 10 days worth of indexed data after
modifying
my config and resetting the cluster. I am not sure exactly how this
happened, and I haven't seen any particular problem in the logs.
I'll
describe my setup and what I did and maybe someone can help shed
some light on what I probably did wrong...
I have a 3 nodes cluster:
node1 is configured as:
http.enabled : true
node.data : false
index.gateway.type : fs
gateway.type : fs
node2 & node3:
http.enabled : false
node.data : true
index.gateway.type : fs
gateway.type : fs
I am using the default index storage settings (which should be niofs
since I am running on Linux) and the default shards and replicas
settings (5 shards and 1 replica).
Basically I changed the config file on my 3 nodes to modify my
custom
stopwords configuration. (and btw, do I need to maintain this
stopwords config on both my data nodes and my http node??). I went
and
did a "bin/service/elasticsearch restart" on my 3 nodes, one after
the
other.
When the cluster was back up, there was a "hole" for the last 10
days
of indexed data. The logs do not show any particular problem.
I have a 3 nodes clusters, with one http+nondata node and two
nonhttp+data nodes.
node 1 config:
http :
enabled : true
node :
data : false
gateway :
type : fs
node 2 & 3
http :
enabled : false
node :
data : true
gateway :
type : fs
So the question is, since this current setup is boggus because the
gateway fs path is not shared between the nodes, what is my best
course of action to correct the problem without loosing my indexes.
Since the data sits on nodes 2 & 3, if I restart node 2 with the new
config, wait for it to get the replication back from node 3 then
proceed with the restart on node 3, wait for replication from node 2
and then finally restart node 1, would that work?
I will either use NFS (I'd rather not) or HDFS (see other thread
"cloud gateway") for the gateway.
Ok, so to play it safe, I'll go with the shared fs gateway for now.
Regarding the config change + required cluster restart, since my
current cluster gateway config is boggus (using local fs storage for
gateway) what would be the best procedure to restart my cluster
hopefully without loosing my current indexes? Again, my setup is 1
non-data+http node and 2 data+non-http nodes.
You don't need to set the index.gateway.type, since it will, as you
pointed
out, inherit the gateway type. Regarding the first setting of shared fs
gateway, you need to set the location on the gateway level:
gateway:
type: fs
fs:
location: /mnt/sharedfs
You should not set the location of the index gateway (its there, but for
really advance use cases). The location on the gateway level is
important
since that where the cluster meta data will be stored.
A disclaimer regarding the cloud gateway, i have hear people having
problem
with it (caused by jclouds). For 0.9 I am going to write a native AWS
cloud
version.
-shay.banon
I will finally go with the S3 cloud gateway using config like:
gateway:
type: cloud
cloud:
chunk_size: 1g
container: mycontainer
index :
gateway :
type : cloud
Now, is the last "index.gateway.type : cloud" config needed or it is
completely redundant to the node level gateway config? What I
understand from the doc is that the index level gateway will inherit
from the node level gateway and index level gateway config is only
required to specify exceptions to the node level gateway config.
Ok, obviously, I did not get that and when I tested my config it was
multiple nodes on a single host...
I will then create a shared fs across my nodes and restart my cluster
to pickup the new config. Is there a way for me to restart my cluster
without loosing my indexes, in the state it is in now? (with a boggus
gateway config).
Yesterday I experienced a strange situation on my ES cluster; I
basically lost the last 10 days worth of indexed data after
modifying
my config and resetting the cluster. I am not sure exactly how this
happened, and I haven't seen any particular problem in the logs.
I'll
describe my setup and what I did and maybe someone can help shed
some light on what I probably did wrong...
I have a 3 nodes cluster:
node1 is configured as:
http.enabled : true
node.data : false
index.gateway.type : fs
gateway.type : fs
node2 & node3:
http.enabled : false
node.data : true
index.gateway.type : fs
gateway.type : fs
I am using the default index storage settings (which should be
niofs
since I am running on Linux) and the default shards and replicas
settings (5 shards and 1 replica).
Basically I changed the config file on my 3 nodes to modify my
custom
stopwords configuration. (and btw, do I need to maintain this
stopwords config on both my data nodes and my http node??). I went
and
did a "bin/service/elasticsearch restart" on my 3 nodes, one after
the
other.
When the cluster was back up, there was a "hole" for the last 10
days
of indexed data. The logs do not show any particular problem.
If you are going to use FS / HDFS, you can use the cluster state API to find
out which node is the primary, then copy over the content of the FS gateway
(under the work directory) and copy it over.
I have a 3 nodes clusters, with one http+nondata node and two
nonhttp+data nodes.
node 1 config:
http :
enabled : true
node :
data : false
gateway :
type : fs
node 2 & 3
http :
enabled : false
node :
data : true
gateway :
type : fs
So the question is, since this current setup is boggus because the
gateway fs path is not shared between the nodes, what is my best
course of action to correct the problem without loosing my indexes.
Since the data sits on nodes 2 & 3, if I restart node 2 with the new
config, wait for it to get the replication back from node 3 then
proceed with the restart on node 3, wait for replication from node 2
and then finally restart node 1, would that work?
I will either use NFS (I'd rather not) or HDFS (see other thread
"cloud gateway") for the gateway.
Ok, so to play it safe, I'll go with the shared fs gateway for now.
Regarding the config change + required cluster restart, since my
current cluster gateway config is boggus (using local fs storage for
gateway) what would be the best procedure to restart my cluster
hopefully without loosing my current indexes? Again, my setup is 1
non-data+http node and 2 data+non-http nodes.
You don't need to set the index.gateway.type, since it will, as you
pointed
out, inherit the gateway type. Regarding the first setting of shared
fs
gateway, you need to set the location on the gateway level:
gateway:
type: fs
fs:
location: /mnt/sharedfs
You should not set the location of the index gateway (its there, but
for
really advance use cases). The location on the gateway level is
important
since that where the cluster meta data will be stored.
A disclaimer regarding the cloud gateway, i have hear people having
problem
with it (caused by jclouds). For 0.9 I am going to write a native AWS
cloud
version.
-shay.banon
I will finally go with the S3 cloud gateway using config like:
gateway:
type: cloud
cloud:
chunk_size: 1g
container: mycontainer
index :
gateway :
type : cloud
Now, is the last "index.gateway.type : cloud" config needed or it is
completely redundant to the node level gateway config? What I
understand from the doc is that the index level gateway will inherit
from the node level gateway and index level gateway config is only
required to specify exceptions to the node level gateway config.
Ok, obviously, I did not get that and when I tested my config it
was
multiple nodes on a single host...
I will then create a shared fs across my nodes and restart my
cluster
to pickup the new config. Is there a way for me to restart my
cluster
without loosing my indexes, in the state it is in now? (with a
boggus
gateway config).
Yesterday I experienced a strange situation on my ES cluster; I
basically lost the last 10 days worth of indexed data after
modifying
my config and resetting the cluster. I am not sure exactly how
this
happened, and I haven't seen any particular problem in the logs.
I'll
describe my setup and what I did and maybe someone can help
shed
some light on what I probably did wrong...
I have a 3 nodes cluster:
node1 is configured as:
http.enabled : true
node.data : false
index.gateway.type : fs
gateway.type : fs
node2 & node3:
http.enabled : false
node.data : true
index.gateway.type : fs
gateway.type : fs
I am using the default index storage settings (which should be
niofs
since I am running on Linux) and the default shards and replicas
settings (5 shards and 1 replica).
Basically I changed the config file on my 3 nodes to modify my
custom
stopwords configuration. (and btw, do I need to maintain this
stopwords config on both my data nodes and my http node??). I
went
and
did a "bin/service/elasticsearch restart" on my 3 nodes, one
after
the
other.
When the cluster was back up, there was a "hole" for the last 10
days
of indexed data. The logs do not show any particular problem.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
can help shed
can help shed