Min Hard Drive Requirements


(Ophir Michaeli) #1

Hi all,

I'm testing the indexing of 100 million documents, it took about 400GB of
the hard drive.
Is there a minimum free hard drive space needed for the index to work OK?
I'm asking because after we indexed 100 million documents we tested the
index and it worked OK,
but then when trying to optimize the optimize took days and then the index
did not respond.
The hard drive had only 10 GB free space so we tried to copy the index to a
new hard drive with a bigger free space, but the index is still not
functioning.

Thank you,
Ophir

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3405d84f-49d4-4cf9-836e-6b6bc09fdc74%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Mark Walkom) #2

It will work until it's full, but then ES will fall over.
Merging does require a certain amount of disk space, usually the same
amount as the segment that is being merged as it has to take a copy of the
shard to work on. So for a 10GB segment, you'd need at least 10GB free.

How many shards do you have for the index, or how many are you trying to
optimise (merge) down to?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 2 July 2014 18:13, Ophir Michaeli ophirmichaeli@gmail.com wrote:

Hi all,

I'm testing the indexing of 100 million documents, it took about 400GB of
the hard drive.
Is there a minimum free hard drive space needed for the index to work OK?
I'm asking because after we indexed 100 million documents we tested the
index and it worked OK,
but then when trying to optimize the optimize took days and then the index
did not respond.
The hard drive had only 10 GB free space so we tried to copy the index to
a new hard drive with a bigger free space, but the index is still not
functioning.

Thank you,
Ophir

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/3405d84f-49d4-4cf9-836e-6b6bc09fdc74%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/3405d84f-49d4-4cf9-836e-6b6bc09fdc74%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624aptU73HX9C3TtkcXULpG6YYF6TzZ7jUZOY8mE6grx8yw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Ophir Michaeli) #3

When I tried to optimize the index had 51 shards.
Regards, Ophir

On Wednesday, July 2, 2014 11:27:50 AM UTC+3, Mark Walkom wrote:

It will work until it's full, but then ES will fall over.
Merging does require a certain amount of disk space, usually the same
amount as the segment that is being merged as it has to take a copy of the
shard to work on. So for a 10GB segment, you'd need at least 10GB free.

How many shards do you have for the index, or how many are you trying to
optimise (merge) down to?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com <javascript:>
web: www.campaignmonitor.com

On 2 July 2014 18:13, Ophir Michaeli <ophirm...@gmail.com <javascript:>>
wrote:

Hi all,

I'm testing the indexing of 100 million documents, it took about 400GB of
the hard drive.
Is there a minimum free hard drive space needed for the index to work OK?
I'm asking because after we indexed 100 million documents we tested the
index and it worked OK,
but then when trying to optimize the optimize took days and then the
index did not respond.
The hard drive had only 10 GB free space so we tried to copy the index to
a new hard drive with a bigger free space, but the index is still not
functioning.

Thank you,
Ophir

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/3405d84f-49d4-4cf9-836e-6b6bc09fdc74%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/3405d84f-49d4-4cf9-836e-6b6bc09fdc74%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6cdd64fb-dfe7-479f-b44c-3c1e8cff1ce7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Mark Walkom) #4

Ok, how many were you reducing to? How big is the index?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 3 July 2014 02:03, Ophir Michaeli ophirmichaeli@gmail.com wrote:

When I tried to optimize the index had 51 shards.
Regards, Ophir

On Wednesday, July 2, 2014 11:27:50 AM UTC+3, Mark Walkom wrote:

It will work until it's full, but then ES will fall over.
Merging does require a certain amount of disk space, usually the same
amount as the segment that is being merged as it has to take a copy of the
shard to work on. So for a 10GB segment, you'd need at least 10GB free.

How many shards do you have for the index, or how many are you trying to
optimise (merge) down to?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 2 July 2014 18:13, Ophir Michaeli ophirm...@gmail.com wrote:

Hi all,

I'm testing the indexing of 100 million documents, it took about 400GB
of the hard drive.
Is there a minimum free hard drive space needed for the index to work OK?
I'm asking because after we indexed 100 million documents we tested the
index and it worked OK,
but then when trying to optimize the optimize took days and then the
index did not respond.
The hard drive had only 10 GB free space so we tried to copy the index
to a new hard drive with a bigger free space, but the index is still not
functioning.

Thank you,
Ophir

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/3405d84f-49d4-4cf9-836e-6b6bc09fdc74%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/3405d84f-49d4-4cf9-836e-6b6bc09fdc74%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/6cdd64fb-dfe7-479f-b44c-3c1e8cff1ce7%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/6cdd64fb-dfe7-479f-b44c-3c1e8cff1ce7%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624aymYEHAGEg4z5XhXxnpgyCgobzPeoC8Wg%3DMB%3DM3AhCMw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(system) #5