I discovered a problem today in my cluster. Some nodes started complaining
that they had no disk space.
On Elasticsearch-head i can only account for 100GB of cluster data. On the
filesystem on the other hand i have found 100GB in "data/cluster/nodes/0"
and 96GB in "data/cluster/nodes/1".
I have no idea where this data in folder "data/cluster/nodes/1" comes from.
I'm looking for information on how can this happen. What can i do to avoid
this happening again. And if I just remove /1 will there be a problem
(besides the probable data loss)?
At one point, you had two instances of elasticsearch running on the same
node. Given the same config, it will create a new subfolder under the data
directory.
You would first need to find out if there are two instances running
currently. If so, you would need to make an educated guess about which one
is in the cluster.
Cheers,
Ivan
On Fri, Oct 11, 2013 at 10:35 AM, MagmaRules mfcoxo@gmail.com wrote:
Hello,
I discovered a problem today in my cluster. Some nodes started complaining
that they had no disk space.
On Elasticsearch-head i can only account for 100GB of cluster data. On the
filesystem on the other hand i have found 100GB in "data/cluster/nodes/0"
and 96GB in "data/cluster/nodes/1".
I have no idea where this data in folder "data/cluster/nodes/1" comes
from.
I'm looking for information on how can this happen. What can i do to avoid
this happening again. And if I just remove /1 will there be a problem
(besides the probable data loss)?
Im going to shutdown my cluster. If i start it up all will be running with
the folder "nodes/0" correct? I can just delete "nodes/1" ?
As a way to stop this from happening again i'm thinking on removing write
privileges on "nodes/1". Will that work or will it just create "nodes/2"?
On Saturday, October 12, 2013 1:03:30 AM UTC+1, Ivan Brusic wrote:
At one point, you had two instances of elasticsearch running on the same
node. Given the same config, it will create a new subfolder under the data
directory.
You would first need to find out if there are two instances running
currently. If so, you would need to make an educated guess about which one
is in the cluster.
Cheers,
Ivan
On Fri, Oct 11, 2013 at 10:35 AM, MagmaRules <mfc...@gmail.com<javascript:>
wrote:
Hello,
I discovered a problem today in my cluster. Some nodes started
complaining that they had no disk space.
On Elasticsearch-head i can only account for 100GB of cluster data. On
the filesystem on the other hand i have found 100GB in
"data/cluster/nodes/0" and 96GB in "data/cluster/nodes/1".
I have no idea where this data in folder "data/cluster/nodes/1" comes
from.
I'm looking for information on how can this happen. What can i do to
avoid this happening again. And if I just remove /1 will there be a problem
(besides the probable data loss)?
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.
Yes, it will create a nodes/2 directly. The best solution would be to use a
script that starts up elasticsearch and checks to see if it is already
running via a pid. There existing service wrapper does that, plus I think
there are now actual *nix distributions.
If only one instance is currently running, then the correct directory is
the one with the newest timestamps. No need to restart the cluster, just
delete to directories.
Cheers,
Ivan
On Sat, Oct 12, 2013 at 3:24 AM, MagmaRules mfcoxo@gmail.com wrote:
Im going to shutdown my cluster. If i start it up all will be running with
the folder "nodes/0" correct? I can just delete "nodes/1" ?
As a way to stop this from happening again i'm thinking on removing write
privileges on "nodes/1". Will that work or will it just create "nodes/2"?
On Saturday, October 12, 2013 1:03:30 AM UTC+1, Ivan Brusic wrote:
At one point, you had two instances of elasticsearch running on the same
node. Given the same config, it will create a new subfolder under the data
directory.
You would first need to find out if there are two instances running
currently. If so, you would need to make an educated guess about which one
is in the cluster.
Cheers,
Ivan
On Fri, Oct 11, 2013 at 10:35 AM, MagmaRules mfc...@gmail.com wrote:
Hello,
I discovered a problem today in my cluster. Some nodes started
complaining that they had no disk space.
On Elasticsearch-head i can only account for 100GB of cluster data. On
the filesystem on the other hand i have found 100GB in
"data/cluster/nodes/0" and 96GB in "data/cluster/nodes/1".
I have no idea where this data in folder "data/cluster/nodes/1" comes
from.
I'm looking for information on how can this happen. What can i do to
avoid this happening again. And if I just remove /1 will there be a problem
(besides the probable data loss)?
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.
I ended up just deleting nodes/1 and restarting the cluster. I lost one
shard that got corrupted because there was no disk space on the master and
the backup.
I did make a backup of this shard from nodes/1 and used it to restore
almost all data from an older version and only lost about 600MB of data.
Still haven't found what allowed the creation of nodes/1.
On Saturday, October 12, 2013 4:21:40 PM UTC+1, Ivan Brusic wrote:
Yes, it will create a nodes/2 directly. The best solution would be to use
a script that starts up elasticsearch and checks to see if it is already
running via a pid. There existing service wrapper does that, plus I think
there are now actual *nix distributions.
If only one instance is currently running, then the correct directory is
the one with the newest timestamps. No need to restart the cluster, just
delete to directories.
Cheers,
Ivan
On Sat, Oct 12, 2013 at 3:24 AM, MagmaRules <mfc...@gmail.com<javascript:>
wrote:
Im going to shutdown my cluster. If i start it up all will be running
with the folder "nodes/0" correct? I can just delete "nodes/1" ?
As a way to stop this from happening again i'm thinking on removing write
privileges on "nodes/1". Will that work or will it just create "nodes/2"?
On Saturday, October 12, 2013 1:03:30 AM UTC+1, Ivan Brusic wrote:
At one point, you had two instances of elasticsearch running on the same
node. Given the same config, it will create a new subfolder under the data
directory.
You would first need to find out if there are two instances running
currently. If so, you would need to make an educated guess about which one
is in the cluster.
Cheers,
Ivan
On Fri, Oct 11, 2013 at 10:35 AM, MagmaRules mfc...@gmail.com wrote:
Hello,
I discovered a problem today in my cluster. Some nodes started
complaining that they had no disk space.
On Elasticsearch-head i can only account for 100GB of cluster data. On
the filesystem on the other hand i have found 100GB in
"data/cluster/nodes/0" and 96GB in "data/cluster/nodes/1".
I have no idea where this data in folder "data/cluster/nodes/1" comes
from.
I'm looking for information on how can this happen. What can i do to
avoid this happening again. And if I just remove /1 will there be a problem
(besides the probable data loss)?
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.