# Lucene vs elasticsearch file size **URL:** \ **Category:** Elasticsearch\ **Created:** [February 27, 2013, 1:26pm UTC](https://discuss.elastic.co/t/lucene-vs-elasticsearch-file-size/10925 "2013-02-27T13:26:06Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Ophir\_Michaeli](https://avatars.discourse-cdn.com/v4/letter/o/9f8e36/32.png) [@Ophir\_Michaeli](https://discuss.elastic.co/u/Ophir_Michaeli)\ **Post date:** [February 27, 2013, 1:26pm UTC](https://discuss.elastic.co/t/lucene-vs-elasticsearch-file-size/10925/1 "2013-02-27T13:26:06Z")
Hi, I ran identical indexing tests for elasticsearch and lucene and the size of the index in elasticsearch is 3 times more than Lucene. What is the explanation for this? We're considering moving from Lucene to elasticsearch and the files size is important for a huge indexing system. Thanks, Ophir -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [February 27, 2013, 2:03pm UTC](https://discuss.elastic.co/t/lucene-vs-elasticsearch-file-size/10925/2 "2013-02-27T14:03:41Z")
Hi Ophir, Do you use all defaults? If so, you should be aware that Elasticsearch stores your index, but also your source document as is and creates one replica of each shard (not sure your are concerned for this if you started a single node). What version do you use? Did you enable compression? Starting from 0.90, compression is now always enabled. [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/store.html) Does it start to answer? -- David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) @dadoonet | @elasticsearchfr | @scrutmydocs Le 27 févr. 2013 à 14:26, Ophir Michaeli [ophirmichaeli@gmail.com](mailto:ophirmichaeli@gmail.com) a écrit : > Hi, > > I ran identical indexing tests for elasticsearch and lucene and the size of the index in elasticsearch is 3 times more than Lucene. > What is the explanation for this? > We're considering moving from Lucene to elasticsearch and the files size is important for a huge indexing system. > > Thanks, > Ophir > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ophir\_Michaeli](https://avatars.discourse-cdn.com/v4/letter/o/9f8e36/32.png) [@Ophir\_Michaeli](https://discuss.elastic.co/u/Ophir_Michaeli)\ **Post date:** [February 27, 2013, 2:38pm UTC](https://discuss.elastic.co/t/lucene-vs-elasticsearch-file-size/10925/3 "2013-02-27T14:38:51Z")
I'm not using defaults, this is the .net Nest code for index creation: MapAndAnalyze( ``` a => a.Analyzers(an => an.Add(snowball, new SnowballAnalyzer{ Language = language })), m => m .Properties(p => p .String(sm => sm .Name(f => ``` f.Description) .IndexAnalyzer(snowball) .SearchAnalyzer(snowball) ``` .Store(false ``` ) ``` ) .String(sm => sm.Name(f => ``` f.Board).Store(false)) ``` .String(sm => sm.Name(f => ``` f.User).Store(false)) ``` .String(sm => sm.Name(f => ``` f.IDPin).Store()) ``` .String(sm => sm.Name(f => ``` f.IDPicture).Store()) ``` ) .DisableAllField() ); ``` I'm not saving "All" field (\_source) and store only 2 fields (as I do with lucene). From what I saw working with 1 replicas and then with 1 shard, the 1 replica was not created till I opened a new node. I'm using elasticsearch version 0.20.2. Is compression default? Thanks! On Wednesday, February 27, 2013 4:03:41 PM UTC+2, David Pilato wrote: > Hi Ophir, > > Do you use all defaults? > > If so, you should be aware that Elasticsearch stores your index, but also > your source document as is and creates one replica of each shard (not sure > your are concerned for this if you started a single node). > > What version do you use? Did you enable compression? Starting from 0.90, > compression is now always enabled. > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/store.html) > > Does it start to answer? > > -- > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)_ > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr[https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) > | @scrutmydocs [https://twitter.com/scrutmydocs](https://twitter.com/scrutmydocs) > > Le 27 févr. 2013 à 14:26, Ophir Michaeli \<[ophirm...@gmail.com](mailto:ophirm...@gmail.com)\\> > a écrit : > > Hi, > > I ran identical indexing tests for elasticsearch and lucene and the size > of the index in elasticsearch is 3 times more than Lucene. > What is the explanation for this? > We're considering moving from Lucene to elasticsearch and the files size > is important for a huge indexing system. > > Thanks, > Ophir > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [February 27, 2013, 3:03pm UTC](https://discuss.elastic.co/t/lucene-vs-elasticsearch-file-size/10925/4 "2013-02-27T15:03:07Z")
\_all and \_source are different fields. > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. Compression before 0.90 is not by default. -- David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) @dadoonet | @elasticsearchfr | @scrutmydocs Le 27 févr. 2013 à 15:38, Ophir Michaeli [ophirmichaeli@gmail.com](mailto:ophirmichaeli@gmail.com) a écrit : > I'm not using defaults, this is the .net Nest code for index creation: > MapAndAnalyze( > a =\> a.Analyzers(an =\> an.Add(snowball, new SnowballAnalyzer { Language = language })), > m =\> m > .Properties(p =\> p > .String(sm =\> sm > .Name(f =\> f.Description) > .IndexAnalyzer(snowball) > .SearchAnalyzer(snowball) > .Store(false) > ) > .String(sm =\> sm.Name(f =\> f.Board).Store(false)) > .String(sm =\> sm.Name(f =\> f.User).Store(false)) > .String(sm =\> sm.Name(f =\> f.IDPin).Store()) > .String(sm =\> sm.Name(f =\> f.IDPicture).Store()) > ) > .DisableAllField() > ); > > I'm not saving "All" field (\_source) and store only 2 fields (as I do with lucene). > From what I saw working with 1 replicas and then with 1 shard, the 1 replica was not created till I opened a new node. > I'm using elasticsearch version 0.20.2. > Is compression default? > Thanks! > On Wednesday, February 27, 2013 4:03:41 PM UTC+2, David Pilato wrote: > Hi Ophir, > > Do you use all defaults? > > If so, you should be aware that Elasticsearch stores your index, but also your source document as is and creates one replica of each shard (not sure your are concerned for this if you started a single node). > > What version do you use? Did you enable compression? Starting from 0.90, compression is now always enabled. [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/store.html) > > Does it start to answer? > > -- > David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) > @dadoonet | @elasticsearchfr | @scrutmydocs > > Le 27 févr. 2013 à 14:26, Ophir Michaeli [ophirm...@gmail.com](mailto:ophirm...@gmail.com) a écrit : > > > Hi, > > > > I ran identical indexing tests for elasticsearch and lucene and the size of the index in elasticsearch is 3 times more than Lucene. > > What is the explanation for this? > > We're considering moving from Lucene to elasticsearch and the files size is important for a huge indexing system. > > > > Thanks, > > Ophir > > > > -- > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![simonw\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simonw_2/32/1130_2.png) [@simonw\_2](https://discuss.elastic.co/u/simonw_2)\ **Post date:** [February 28, 2013, 8:07am UTC](https://discuss.elastic.co/t/lucene-vs-elasticsearch-file-size/10925/5 "2013-02-28T08:07:06Z")
Hey Ophir, as David said, you should make sure you disable the \_source field otherwise ES will store the actual json used to create (index) your document. If you run those kind of comparisons make sure you optimize the index afterwards (force merge) otherwise your segments might be of different size. simon On Wednesday, February 27, 2013 4:03:07 PM UTC+1, David Pilato wrote: > \_all and \_source are different fields. > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/mapping/all-field.html) > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/mapping/source-field.html) > > Compression before 0.90 is not by default. > > -- > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)_ > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr[https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) > | @scrutmydocs [https://twitter.com/scrutmydocs](https://twitter.com/scrutmydocs) > > Le 27 févr. 2013 à 15:38, Ophir Michaeli \<[ophirm...@gmail.com](mailto:ophirm...@gmail.com)\\> > a écrit : > > I'm not using defaults, this is the .net Nest code for index creation: > MapAndAnalyze( > a =\> a.Analyzers(an =\> an.Add(snowball, new > SnowballAnalyzer { Language = language })), > m =\> m > .Properties(p =\> p > .String(sm =\> sm > .Name(f > =\> f.Description) > > .IndexAnalyzer(snowball) > > .SearchAnalyzer(snowball) > .Store( > false) > ) > .String(sm =\> sm.Name(f =\> > f.Board).Store(false)) > .String(sm =\> sm.Name(f =\> > f.User).Store(false)) > .String(sm =\> sm.Name(f =\> > f.IDPin).Store()) > .String(sm =\> sm.Name(f =\> > f.IDPicture).Store()) > ) > .DisableAllField() > ); > > I'm not saving "All" field (\_source) and store only 2 fields (as I do with > lucene). > From what I saw working with 1 replicas and then with 1 shard, the 1 > replica was not created till I opened a new node. > I'm using elasticsearch version 0.20.2. > Is compression default? > Thanks! > On Wednesday, February 27, 2013 4:03:41 PM UTC+2, David Pilato wrote: > > > Hi Ophir, > > > > Do you use all defaults? > > > > If so, you should be aware that Elasticsearch stores your index, but also > > your source document as is and creates one replica of each shard (not sure > > your are concerned for this if you started a single node). > > > > What version do you use? Did you enable compression? Starting from 0.90, > > compression is now always enabled. > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/store.html) > > > > Does it start to answer? > > > > -- > > _David Pilato_ | _Technical Advocate_ | \*[Elasticsearch.com](http://Elasticsearch.com)[http://elasticsearch.com/](http://elasticsearch.com/) > > \* > > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr[https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) > > | @scrutmydocs [https://twitter.com/scrutmydocs](https://twitter.com/scrutmydocs) > > > > Le 27 févr. 2013 à 14:26, Ophir Michaeli [ophirm...@gmail.com](mailto:ophirm...@gmail.com) a écrit : > > > > Hi, > > > > I ran identical indexing tests for elasticsearch and lucene and the size > > of the index in elasticsearch is 3 times more than Lucene. > > What is the explanation for this? > > We're considering moving from Lucene to elasticsearch and the files size > > is important for a huge indexing system. > > > > Thanks, > > Ophir > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:49am UTC](https://discuss.elastic.co/t/lucene-vs-elasticsearch-file-size/10925/6 "2017-07-06T02:49:06Z")