# How to optimize disk usage? **URL:** \ **Category:** Elasticsearch\ **Created:** [May 18, 2012, 6:41pm UTC](https://discuss.elastic.co/t/how-to-optimize-disk-usage/7767 "2012-05-18T18:41:04Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![llowder](https://avatars.discourse-cdn.com/v4/letter/l/e79b87/32.png) [@llowder](https://discuss.elastic.co/u/llowder)\ **Post date:** [May 18, 2012, 6:41pm UTC](https://discuss.elastic.co/t/how-to-optimize-disk-usage/7767/1 "2012-05-18T18:41:04Z")
Currently testing out a program that uses ElasticSearch to store large amounts of data (from log files). My initial test showed a 250% increase in disk usage from the raw data to what was stored in ES. I have looked online some, but am new enough to this that I may not have used the right terms. What options are available to to optimize/tune disk usage when using ES? Can someone please either give me some tips or point me to some documents? Mostly what I will be storing is logs from tomcat, apache and 70+ web apps. Thank you. ---
**Author:** ![Paul\_Brown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_brown/32/1449_2.png) [@Paul\_Brown](https://discuss.elastic.co/u/Paul_Brown)\ **Post date:** [May 18, 2012, 7:41pm UTC](https://discuss.elastic.co/t/how-to-optimize-disk-usage/7767/2 "2012-05-18T19:41:50Z")
This sounds reasonable to me. If you have replicas and indexes and are storing most of your data, then 2.5x is reasonable. (Index ~ data/3, so 2 x (data + data/3) ~2.5x data.) -- Paul On May 18, 2012, at 1:41 PM, [llowder@oreillyauto.com](mailto:llowder@oreillyauto.com) wrote: > Currently testing out a program that uses Elasticsearch to store large amounts of data (from log files). > > My initial test showed a 250% increase in disk usage from the raw data to what was stored in ES. > > I have looked online some, but am new enough to this that I may not have used the right terms. > > What options are available to to optimize/tune disk usage when using ES? Can someone please either give me some tips or point me to some documents? > > Mostly what I will be storing is logs from tomcat, apache and 70+ web apps. > > Thank you. ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [May 19, 2012, 1:08am UTC](https://discuss.elastic.co/t/how-to-optimize-disk-usage/7767/3 "2012-05-19T01:08:51Z")
Hi, Things to look into: - number of replicas. Set to 0 and then compare sizes. - \_source is on and uncompressed? - \_all is on? - individual fields are marked as stored or just indexed? ## Otis Performance Monitoring for Solr / Elasticsearch / HBase - > **[Sematext Monitoring | Infrastructure Monitoring Service](https://sematext.com/spm)** > > Bring together your servers, Apps, Metrics, Logs & Events ✓ Over 40 integrations to easily collect metrics & events across your whole stack ✓ Start now! On Friday, May 18, 2012 2:41:04 PM UTC-4, [llo...@oreillyauto.com](mailto:llo...@oreillyauto.com) wrote: > Currently testing out a program that uses Elasticsearch to store large > amounts of data (from log files). > > My initial test showed a 250% increase in disk usage from the raw data to > what was stored in ES. > > I have looked online some, but am new enough to this that I may not have > used the right terms. > > What options are available to to optimize/tune disk usage when using ES? > Can someone please either give me some tips or point me to some documents? > > Mostly what I will be storing is logs from tomcat, apache and 70+ web apps. > > Thank you. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [May 20, 2012, 10:03pm UTC](https://discuss.elastic.co/t/how-to-optimize-disk-usage/7767/4 "2012-05-20T22:03:56Z")
Here is some info on the logstash wiki on how to reduce the storage: > **[GitHub - elastic/logstash: Logstash - transport and process your logs,...](https://github.com/elastic/logstash)** > > Logstash - transport and process your logs, events, or other data - GitHub - elastic/logstash: Logstash - transport and process your logs, events, or other data . On Fri, May 18, 2012 at 8:41 PM, [llowder@oreillyauto.com](mailto:llowder@oreillyauto.com) \< [llowder@oreillyauto.com](mailto:llowder@oreillyauto.com)\> wrote: > Currently testing out a program that uses Elasticsearch to store large > amounts of data (from log files). > > My initial test showed a 250% increase in disk usage from the raw data to > what was stored in ES. > > I have looked online some, but am new enough to this that I may not have > used the right terms. > > What options are available to to optimize/tune disk usage when using ES? > Can someone please either give me some tips or point me to some documents? > > Mostly what I will be storing is logs from tomcat, apache and 70+ web apps. > > Thank you. ---
**Author:** ![Eric\_Jain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eric_jain/32/834_2.png) [@Eric\_Jain](https://discuss.elastic.co/u/Eric_Jain)\ **Post date:** [May 21, 2012, 4:51pm UTC](https://discuss.elastic.co/t/how-to-optimize-disk-usage/7767/5 "2012-05-21T16:51:44Z")
On May 18, 11:41 am, "[llow...@oreillyauto.com](mailto:llow...@oreillyauto.com)" [llow...@oreillyauto.com](mailto:llow...@oreillyauto.com) wrote: > Can someone please either give me some tips or point me to some documents? You can use Luke [[Google Code Archive - Long-term storage for Google Code Project Hosting.](http://code.google.com/p/luke/)] to peek inside the index and see if there is anything unnecessary in there. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:27am UTC](https://discuss.elastic.co/t/how-to-optimize-disk-usage/7767/6 "2017-07-06T03:27:47Z")