3 nodes to a cluster and 1 using a lot more disk space


(Mike Zupan) #1

I have 3 nodes in a cluster and 1 (the primary right now) is using a lot
more disk space then the other 2. Is this normal?

/dev/mapper/data--group-data--vol 50G 32G 15G 69% /data

vs

/dev/mapper/data--group-data--vol 50G 18G 30G 38% /data
/dev/mapper/data--group-data--vol 50G 19G 29G 40% /data

There is nothing else using this partition except for elasticsearch.

Thanks

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Ivan Brusic) #2

Out of curiosity, under /dev/mapper/data--group-data--vol/data/${
cluster.name}/nodes, how many directories are there? How about just under
/dev/mapper/data--group-data--vol/data/? Perhaps you have some old data
under a different cluster name, or you started two instances on the same
node resulting in two directories.

Cheers,

Ivan

On Tue, Nov 12, 2013 at 11:26 AM, Mike Zupan mike@recurly.com wrote:

I have 3 nodes in a cluster and 1 (the primary right now) is using a lot
more disk space then the other 2. Is this normal?

/dev/mapper/data--group-data--vol 50G 32G 15G 69% /data

vs

/dev/mapper/data--group-data--vol 50G 18G 30G 38% /data
/dev/mapper/data--group-data--vol 50G 19G 29G 40% /data

There is nothing else using this partition except for elasticsearch.

Thanks

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Mike Zupan) #3

Ivan,

Thanks for the reply.. here they are for the 2 that are the same for
/data/cluster.name/nodes

$ ls
0
$
0

Then for the one larger

$ ls
0 1

So it seems like you are correct.

Then on the larger one

du -h --max-depth=1

2.1G ./1
14G ./0

So it seems there is an extra 1 that might have a lot of data.

On Tuesday, November 12, 2013 7:48:48 PM UTC-8, Ivan Brusic wrote:

Out of curiosity, under /dev/mapper/data--group-data--vol/data/${
cluster.name}/nodes, how many directories are there? How about just under
/dev/mapper/data--group-data--vol/data/? Perhaps you have some old data
under a different cluster name, or you started two instances on the same
node resulting in two directories.

Cheers,

Ivan

On Tue, Nov 12, 2013 at 11:26 AM, Mike Zupan <mi...@recurly.com<javascript:>

wrote:

I have 3 nodes in a cluster and 1 (the primary right now) is using a lot
more disk space then the other 2. Is this normal?

/dev/mapper/data--group-data--vol 50G 32G 15G 69% /data

vs

/dev/mapper/data--group-data--vol 50G 18G 30G 38% /data
/dev/mapper/data--group-data--vol 50G 19G 29G 40% /data

There is nothing else using this partition except for elasticsearch.

Thanks

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Mike Zupan) #4

So it seems the data is all in

16G ./elasticsearch/nodes/0

That is under ./data/

On Tuesday, November 12, 2013 8:36:23 PM UTC-8, Mike Zupan wrote:

Ivan,

Thanks for the reply.. here they are for the 2 that are the same for /data/
cluster.name/nodes

$ ls
0
$
0

Then for the one larger

$ ls
0 1

So it seems like you are correct.

Then on the larger one

du -h --max-depth=1

2.1G ./1
14G ./0

So it seems there is an extra 1 that might have a lot of data.

On Tuesday, November 12, 2013 7:48:48 PM UTC-8, Ivan Brusic wrote:

Out of curiosity, under /dev/mapper/data--group-data--vol/data/${
cluster.name}/nodes, how many directories are there? How about just
under /dev/mapper/data--group-data--vol/data/? Perhaps you have some old
data under a different cluster name, or you started two instances on the
same node resulting in two directories.

Cheers,

Ivan

On Tue, Nov 12, 2013 at 11:26 AM, Mike Zupan mi...@recurly.com wrote:

I have 3 nodes in a cluster and 1 (the primary right now) is using a lot
more disk space then the other 2. Is this normal?

/dev/mapper/data--group-data--vol 50G 32G 15G 69% /data

vs

/dev/mapper/data--group-data--vol 50G 18G 30G 38% /data
/dev/mapper/data--group-data--vol 50G 19G 29G 40% /data

There is nothing else using this partition except for elasticsearch.

Thanks

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Ivan Brusic) #5

So at one point you must have started two instances on the same node. Try
using the server wrapper or other method (rpm, etc..) that uses a pid to
only run one instance at a time.

--
Ivan

On Tue, Nov 12, 2013 at 8:38 PM, Mike Zupan mike@recurly.com wrote:

So it seems the data is all in

16G ./elasticsearch/nodes/0

That is under ./data/

On Tuesday, November 12, 2013 8:36:23 PM UTC-8, Mike Zupan wrote:

Ivan,

Thanks for the reply.. here they are for the 2 that are the same for
/data/cluster.name/nodes

$ ls
0
$
0

Then for the one larger

$ ls
0 1

So it seems like you are correct.

Then on the larger one

du -h --max-depth=1

2.1G ./1
14G ./0

So it seems there is an extra 1 that might have a lot of data.

On Tuesday, November 12, 2013 7:48:48 PM UTC-8, Ivan Brusic wrote:

Out of curiosity, under /dev/mapper/data--group-data--vol/data/${
cluster.name}/nodes, how many directories are there? How about just
under /dev/mapper/data--group-data--vol/data/? Perhaps you have some
old data under a different cluster name, or you started two instances on
the same node resulting in two directories.

Cheers,

Ivan

On Tue, Nov 12, 2013 at 11:26 AM, Mike Zupan mi...@recurly.com wrote:

I have 3 nodes in a cluster and 1 (the primary right now) is using a
lot more disk space then the other 2. Is this normal?

/dev/mapper/data--group-data--vol 50G 32G 15G 69% /data

vs

/dev/mapper/data--group-data--vol 50G 18G 30G 38% /data
/dev/mapper/data--group-data--vol 50G 19G 29G 40% /data

There is nothing else using this partition except for elasticsearch.

Thanks

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Mike Zupan) #6

Is there a safe way to clean it?

--
Mike Zupan

On Tuesday, November 12, 2013 at 9:19 PM, Ivan Brusic wrote:

So at one point you must have started two instances on the same node. Try using the server wrapper or other method (rpm, etc..) that uses a pid to only run one instance at a time.

--
Ivan

On Tue, Nov 12, 2013 at 8:38 PM, Mike Zupan <mike@recurly.com (mailto:mike@recurly.com)> wrote:

So it seems the data is all in

16G ./elasticsearch/nodes/0

That is under ./data/

On Tuesday, November 12, 2013 8:36:23 PM UTC-8, Mike Zupan wrote:

Ivan,

Thanks for the reply.. here they are for the 2 that are the same for /data/cluster.name/nodes (http://cluster.name/nodes)

$ ls
0
$
0

Then for the one larger

$ ls
0 1

So it seems like you are correct.

Then on the larger one

du -h --max-depth=1

2.1G ./1
14G ./0

So it seems there is an extra 1 that might have a lot of data.
On Tuesday, November 12, 2013 7:48:48 PM UTC-8, Ivan Brusic wrote:

Out of curiosity, under /dev/mapper/data--group-data--vol/data/${cluster.name (http://cluster.name)}/nodes, how many directories are there? How about just under /dev/mapper/data--group-data--vol/data/? Perhaps you have some old data under a different cluster name, or you started two instances on the same node resulting in two directories.

Cheers,

Ivan

On Tue, Nov 12, 2013 at 11:26 AM, Mike Zupan mi...@recurly.com wrote:

I have 3 nodes in a cluster and 1 (the primary right now) is using a lot more disk space then the other 2. Is this normal?

/dev/mapper/data--group-data--vol 50G 32G 15G 69% /data

vs

/dev/mapper/data--group-data--vol 50G 18G 30G 38% /data
/dev/mapper/data--group-data--vol 50G 19G 29G 40% /data

There is nothing else using this partition except for elasticsearch.

Thanks

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com (mailto:elasticsearch%2Bunsubscribe@googlegroups.com).
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/_EY3rkka4r4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com (mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Ivan Brusic) #7

If you are sure that the files are not in used (timestamp?), you can delete
them directly in the OS.

--
Ivan

On Tue, Nov 12, 2013 at 9:21 PM, Mike Zupan mike@recurly.com wrote:

Is there a safe way to clean it?

--
Mike Zupan

On Tuesday, November 12, 2013 at 9:19 PM, Ivan Brusic wrote:

So at one point you must have started two instances on the same node. Try
using the server wrapper or other method (rpm, etc..) that uses a pid to
only run one instance at a time.

--
Ivan

On Tue, Nov 12, 2013 at 8:38 PM, Mike Zupan mike@recurly.com wrote:

So it seems the data is all in

16G ./elasticsearch/nodes/0

That is under ./data/

On Tuesday, November 12, 2013 8:36:23 PM UTC-8, Mike Zupan wrote:

Ivan,

Thanks for the reply.. here they are for the 2 that are the same for /data/
cluster.name/nodes

$ ls
0
$
0

Then for the one larger

$ ls
0 1

So it seems like you are correct.

Then on the larger one

du -h --max-depth=1

2.1G ./1
14G ./0

So it seems there is an extra 1 that might have a lot of data.

On Tuesday, November 12, 2013 7:48:48 PM UTC-8, Ivan Brusic wrote:

Out of curiosity, under /dev/mapper/data--group-data--vol/data/${
cluster.name}/nodes, how many directories are there? How about just under
/dev/mapper/data--group-data--vol/data/? Perhaps you have some old data
under a different cluster name, or you started two instances on the same
node resulting in two directories.

Cheers,

Ivan

On Tue, Nov 12, 2013 at 11:26 AM, Mike Zupan mi...@recurly.com wrote:

I have 3 nodes in a cluster and 1 (the primary right now) is using a lot
more disk space then the other 2. Is this normal?

/dev/mapper/data--group-data--vol 50G 32G 15G 69% /data

vs

/dev/mapper/data--group-data--vol 50G 18G 30G 38% /data
/dev/mapper/data--group-data--vol 50G 19G 29G 40% /data

There is nothing else using this partition except for elasticsearch.

Thanks

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/_EY3rkka4r4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(system) #8