Compression Mechanism


(hiveesuser) #1

I am using elasticsearch as medium to store my data what is the compression
meachnism used in storing the data.

The reason why I am asking is if I have 50 GB of raw data coming in every
day how much should I plan for my hardware diskspace that's the reason I
want to know how this one works

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/18648a47-04df-4ae0-8f52-98d61d1f6d6a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Mark Walkom) #2

Data use will depend on your indexing rate and how big your documents are,
which is entirely dependant on your use.

However ES uses the lz4 compression algorithm.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 11 August 2014 16:01, hiveesuser@gmail.com wrote:

I am using elasticsearch as medium to store my data what is the
compression meachnism used in storing the data.

The reason why I am asking is if I have 50 GB of raw data coming in every
day how much should I plan for my hardware diskspace that's the reason I
want to know how this one works

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/18648a47-04df-4ae0-8f52-98d61d1f6d6a%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/18648a47-04df-4ae0-8f52-98d61d1f6d6a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624a28pzaYBJHT0j5PU0pRtmxN%2BBkZfksv-JrM2To285Crg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Telax) #3

ElasticSearch uses LZF on stored fields (including _source). The storage requirements will depend on your implementation and the complexity of your data, however, planning for a 1:1 ratio +10% with compression enabled aught to put you on the right path. Otherwise, you'll have to experiment to find out exactly what you'll require.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/64ccbc7e-0fb9-4c7b-ae23-8c69cbd32ca4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Jörg Prante) #4

Elasticsearch uses LZ4, see

For storage requirements, you need around twice the disk space if you
incrementally grow your index, because of additional segment merge space
overhead.

Jörg

On Tue, Aug 12, 2014 at 8:49 AM, Telax andrew.w.nixon@gmail.com wrote:

ElasticSearch uses LZF on stored fields (including _source). The storage
requirements will depend on your implementation and the complexity of your
data, however, planning for a 1:1 ratio +10% with compression enabled aught
to put you on the right path. Otherwise, you'll have to experiment to find
out exactly what you'll require.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/64ccbc7e-0fb9-4c7b-ae23-8c69cbd32ca4%40googlegroups.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEu8aVh%2BiSGXwfUutrTRHJ4cQzzW2-36OpVP3-nd%2B200Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(system) #5