# Elasticsearch index MUCH larger then similar lucene index **URL:** \ **Category:** Elasticsearch\ **Created:** [May 21, 2013, 3:53pm UTC](https://discuss.elastic.co/t/elasticsearch-index-much-larger-then-similar-lucene-index/12056 "2013-05-21T15:53:30Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![shlomivaknin](https://avatars.discourse-cdn.com/v4/letter/s/4da419/32.png) [@shlomivaknin](https://discuss.elastic.co/u/shlomivaknin)\ **Post date:** [May 21, 2013, 3:53pm UTC](https://discuss.elastic.co/t/elasticsearch-index-much-larger-then-similar-lucene-index/12056/1 "2013-05-21T15:53:30Z")
Hey, We have some old java code that uses lucene and grizzly to serve queries over text. we have two field, a string field and a numeric (long) field. the indexing code is pretty straight forward. I was trying to migrate this to elastic, pretty simple configuration, and indexed the same data. the java based implementation took about 6gb, while to elastic took 17gb.. does this makes sense? what could i do about this? Thanks! -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Itamar\_Syn\_Hershko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itamar_syn_hershko/32/725_2.png) [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)\ **Post date:** [May 21, 2013, 3:54pm UTC](https://discuss.elastic.co/t/elasticsearch-index-much-larger-then-similar-lucene-index/12056/2 "2013-05-21T15:54:38Z")
Yes, because ES stores the entire source by default On Tue, May 21, 2013 at 6:53 PM, Shlomi [shlomivaknin@gmail.com](mailto:shlomivaknin@gmail.com) wrote: > Hey, > > We have some old java code that uses lucene and grizzly to serve queries > over text. we have two field, a string field and a numeric (long) field. > the indexing code is pretty straight forward. > > I was trying to migrate this to elastic, pretty simple configuration, and > indexed the same data. > > the java based implementation took about 6gb, while to elastic took 17gb.. > > does this makes sense? what could i do about this? > > Thanks! > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![shlomivaknin](https://avatars.discourse-cdn.com/v4/letter/s/4da419/32.png) [@shlomivaknin](https://discuss.elastic.co/u/shlomivaknin)\ **Post date:** [May 21, 2013, 3:59pm UTC](https://discuss.elastic.co/t/elasticsearch-index-much-larger-then-similar-lucene-index/12056/3 "2013-05-21T15:59:36Z")
yes, so i was trying to exclude source, but then queries didnt return anything besides id. but in any case, even disabling source still gave me a large index.. any way to tell it to save just the fields? On Tuesday, May 21, 2013 6:54:38 PM UTC+3, Itamar Syn-Hershko wrote: > Yes, because ES stores the entire source by default > > On Tue, May 21, 2013 at 6:53 PM, Shlomi \<[shlomi...@gmail.com](mailto:shlomi...@gmail.com) \ > > > wrote: > > > Hey, > > > > We have some old java code that uses lucene and grizzly to serve queries > > over text. we have two field, a string field and a numeric (long) field. > > the indexing code is pretty straight forward. > > > > I was trying to migrate this to elastic, pretty simple configuration, and > > indexed the same data. > > > > the java based implementation took about 6gb, while to elastic took > > 17gb.. > > > > does this makes sense? what could i do about this? > > > > Thanks! > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Michael\_Sick](https://avatars.discourse-cdn.com/v4/letter/m/22d042/32.png) [@Michael\_Sick](https://discuss.elastic.co/u/Michael_Sick)\ **Post date:** [May 21, 2013, 4:01pm UTC](https://discuss.elastic.co/t/elasticsearch-index-much-larger-then-similar-lucene-index/12056/4 "2013-05-21T16:01:46Z")
do you have replication on? On May 21, 2013 9:59 AM, "Shlomi" [shlomivaknin@gmail.com](mailto:shlomivaknin@gmail.com) wrote: > yes, so i was trying to exclude source, but then queries didnt return > anything besides id. but in any case, even disabling source still gave me a > large index.. > > any way to tell it to save just the fields? > > On Tuesday, May 21, 2013 6:54:38 PM UTC+3, Itamar Syn-Hershko wrote: > > > Yes, because ES stores the entire source by default > > > > On Tue, May 21, 2013 at 6:53 PM, Shlomi [shlomi...@gmail.com](mailto:shlomi...@gmail.com) wrote: > > > > > Hey, > > > > > > We have some old java code that uses lucene and grizzly to serve queries > > > over text. we have two field, a string field and a numeric (long) field. > > > the indexing code is pretty straight forward. > > > > > > I was trying to migrate this to elastic, pretty simple configuration, > > > and indexed the same data. > > > > > > the java based implementation took about 6gb, while to elastic took > > > 17gb.. > > > > > > does this makes sense? what could i do about this? > > > > > > Thanks! > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > . > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Itamar\_Syn\_Hershko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itamar_syn_hershko/32/725_2.png) [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)\ **Post date:** [May 21, 2013, 4:02pm UTC](https://discuss.elastic.co/t/elasticsearch-index-much-larger-then-similar-lucene-index/12056/5 "2013-05-21T16:02:46Z")
You can disable storing the source, but then you don't have stored fields unless you specify you want that explicitly. And it costs more to load several fields from store when you don't have source enabled. On Tue, May 21, 2013 at 6:59 PM, Shlomi [shlomivaknin@gmail.com](mailto:shlomivaknin@gmail.com) wrote: > yes, so i was trying to exclude source, but then queries didnt return > anything besides id. but in any case, even disabling source still gave me a > large index.. > > any way to tell it to save just the fields? > > On Tuesday, May 21, 2013 6:54:38 PM UTC+3, Itamar Syn-Hershko wrote: > > > Yes, because ES stores the entire source by default > > > > On Tue, May 21, 2013 at 6:53 PM, Shlomi [shlomi...@gmail.com](mailto:shlomi...@gmail.com) wrote: > > > > > Hey, > > > > > > We have some old java code that uses lucene and grizzly to serve queries > > > over text. we have two field, a string field and a numeric (long) field. > > > the indexing code is pretty straight forward. > > > > > > I was trying to migrate this to elastic, pretty simple configuration, > > > and indexed the same data. > > > > > > the java based implementation took about 6gb, while to elastic took > > > 17gb.. > > > > > > does this makes sense? what could i do about this? > > > > > > Thanks! > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > . > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![mattweber](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mattweber/32/44940_2.png) [@mattweber](https://discuss.elastic.co/u/mattweber)\ **Post date:** [May 21, 2013, 4:03pm UTC](https://discuss.elastic.co/t/elasticsearch-index-much-larger-then-similar-lucene-index/12056/6 "2013-05-21T16:03:19Z")
Don't forget about the \_all field. Also, if you don't store the source, you need to explicitly set "store" to yes on your field mappings so you can have them returned in the results. On Tue, May 21, 2013 at 8:59 AM, Shlomi [shlomivaknin@gmail.com](mailto:shlomivaknin@gmail.com) wrote: > yes, so i was trying to exclude source, but then queries didnt return > anything besides id. but in any case, even disabling source still gave me a > large index.. > > any way to tell it to save just the fields? > > On Tuesday, May 21, 2013 6:54:38 PM UTC+3, Itamar Syn-Hershko wrote: > > > Yes, because ES stores the entire source by default > > > > On Tue, May 21, 2013 at 6:53 PM, Shlomi [shlomi...@gmail.com](mailto:shlomi...@gmail.com) wrote: > > > > > Hey, > > > > > > We have some old java code that uses lucene and grizzly to serve queries > > > over text. we have two field, a string field and a numeric (long) field. > > > the indexing code is pretty straight forward. > > > > > > I was trying to migrate this to elastic, pretty simple configuration, > > > and indexed the same data. > > > > > > the java based implementation took about 6gb, while to elastic took > > > 17gb.. > > > > > > does this makes sense? what could i do about this? > > > > > > Thanks! > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > . > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![shlomivaknin](https://avatars.discourse-cdn.com/v4/letter/s/4da419/32.png) [@shlomivaknin](https://discuss.elastic.co/u/shlomivaknin)\ **Post date:** [May 21, 2013, 4:07pm UTC](https://discuss.elastic.co/t/elasticsearch-index-much-larger-then-similar-lucene-index/12056/7 "2013-05-21T16:07:44Z")
Hey, thanks all, let me reply: Michael - no, i set replicas to 0 (if that what you meant..) Itamar & Matt - i disabled \_all and \_source, and explicitly set "store" to "yes" for both fields (i dont care about perf for now..) - with this setting i still got a much larger size and was still unable to see the fields (although i set store to yes) through queries (only got id's back) On Tuesday, May 21, 2013 7:03:19 PM UTC+3, Matt Weber wrote: > Don't forget about the \_all field. Also, if you don't store the source, > you need to explicitly set "store" to yes on your field mappings so you can > have them returned in the results. > > On Tue, May 21, 2013 at 8:59 AM, Shlomi \<[shlomi...@gmail.com](mailto:shlomi...@gmail.com) \ > > > wrote: > > > yes, so i was trying to exclude source, but then queries didnt return > > anything besides id. but in any case, even disabling source still gave me a > > large index.. > > > > any way to tell it to save just the fields? > > > > On Tuesday, May 21, 2013 6:54:38 PM UTC+3, Itamar Syn-Hershko wrote: > > > > > Yes, because ES stores the entire source by default > > > > > > On Tue, May 21, 2013 at 6:53 PM, Shlomi [shlomi...@gmail.com](mailto:shlomi...@gmail.com) wrote: > > > > > > > Hey, > > > > > > > > We have some old java code that uses lucene and grizzly to serve > > > > queries over text. we have two field, a string field and a numeric (long) > > > > field. the indexing code is pretty straight forward. > > > > > > > > I was trying to migrate this to elastic, pretty simple configuration, > > > > and indexed the same data. > > > > > > > > the java based implementation took about 6gb, while to elastic took > > > > 17gb.. > > > > > > > > does this makes sense? what could i do about this? > > > > > > > > Thanks! > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > > > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > > . > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![shlomivaknin](https://avatars.discourse-cdn.com/v4/letter/s/4da419/32.png) [@shlomivaknin](https://discuss.elastic.co/u/shlomivaknin)\ **Post date:** [May 21, 2013, 4:10pm UTC](https://discuss.elastic.co/t/elasticsearch-index-much-larger-then-similar-lucene-index/12056/8 "2013-05-21T16:10:07Z")
here is a fraction of the mapping i have (i use clojure so its a bit different from json, but its essentially the same): ``` {:test { :_source {:enabled "false" } :_all {:enabled "false" } :properties {:gram {:type "string" :store "yes" ``` :analyzer :ngram-index :compress "true"} :freq {:type "long" :store "yes"} }}}] On Tuesday, May 21, 2013 7:07:44 PM UTC+3, Shlomi wrote: > Hey, > > thanks all, let me reply: > > Michael - no, i set replicas to 0 (if that what you meant..) > > Itamar & Matt - i disabled \_all and \_source, and explicitly set "store" to > "yes" for both fields (i dont care about perf for now..) - with this > setting i still got a much larger size and was still unable to see the > fields (although i set store to yes) through queries (only got id's back) > > On Tuesday, May 21, 2013 7:03:19 PM UTC+3, Matt Weber wrote: > > > Don't forget about the \_all field. Also, if you don't store the source, > > you need to explicitly set "store" to yes on your field mappings so you can > > have them returned in the results. > > > > On Tue, May 21, 2013 at 8:59 AM, Shlomi [shlomi...@gmail.com](mailto:shlomi...@gmail.com) wrote: > > > > > yes, so i was trying to exclude source, but then queries didnt return > > > anything besides id. but in any case, even disabling source still gave me a > > > large index.. > > > > > > any way to tell it to save just the fields? > > > > > > On Tuesday, May 21, 2013 6:54:38 PM UTC+3, Itamar Syn-Hershko wrote: > > > > > > > Yes, because ES stores the entire source by default > > > > > > > > On Tue, May 21, 2013 at 6:53 PM, Shlomi [shlomi...@gmail.com](mailto:shlomi...@gmail.com) wrote: > > > > > > > > > Hey, > > > > > > > > > > We have some old java code that uses lucene and grizzly to serve > > > > > queries over text. we have two field, a string field and a numeric (long) > > > > > field. the indexing code is pretty straight forward. > > > > > > > > > > I was trying to migrate this to elastic, pretty simple configuration, > > > > > and indexed the same data. > > > > > > > > > > the java based implementation took about 6gb, while to elastic took > > > > > 17gb.. > > > > > > > > > > does this makes sense? what could i do about this? > > > > > > > > > > Thanks! > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > > > > > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > > > . > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![shlomivaknin](https://avatars.discourse-cdn.com/v4/letter/s/4da419/32.png) [@shlomivaknin](https://discuss.elastic.co/u/shlomivaknin)\ **Post date:** [May 22, 2013, 9:08am UTC](https://discuss.elastic.co/t/elasticsearch-index-much-larger-then-similar-lucene-index/12056/9 "2013-05-22T09:08:31Z")
does ES store its numeric fields as strings? can someone confirm that if you disable \_source and keep each field as stored and indexed, your fields becomes invisible (although queriable)? or am i doing something totally wrong?.. Thanks On Tuesday, May 21, 2013 7:10:07 PM UTC+3, Shlomi wrote: > here is a fraction of the mapping i have (i use clojure so its a bit > different from json, but its essentially the same): > > ``` > {:test { > :_source {:enabled "false" } > :_all {:enabled "false" } > :properties {:gram {:type "string" :store "yes" > > ``` > > :analyzer :ngram-index :compress "true"} > :freq {:type "long" :store "yes"} > }}}] > > On Tuesday, May 21, 2013 7:07:44 PM UTC+3, Shlomi wrote: > > > Hey, > > > > thanks all, let me reply: > > > > Michael - no, i set replicas to 0 (if that what you meant..) > > > > Itamar & Matt - i disabled \_all and \_source, and explicitly set "store" > > to "yes" for both fields (i dont care about perf for now..) - with this > > setting i still got a much larger size and was still unable to see the > > fields (although i set store to yes) through queries (only got id's back) > > > > On Tuesday, May 21, 2013 7:03:19 PM UTC+3, Matt Weber wrote: > > > > > Don't forget about the \_all field. Also, if you don't store the source, > > > you need to explicitly set "store" to yes on your field mappings so you can > > > have them returned in the results. > > > > > > On Tue, May 21, 2013 at 8:59 AM, Shlomi [shlomi...@gmail.com](mailto:shlomi...@gmail.com) wrote: > > > > > > > yes, so i was trying to exclude source, but then queries didnt return > > > > anything besides id. but in any case, even disabling source still gave me a > > > > large index.. > > > > > > > > any way to tell it to save just the fields? > > > > > > > > On Tuesday, May 21, 2013 6:54:38 PM UTC+3, Itamar Syn-Hershko wrote: > > > > > > > > > Yes, because ES stores the entire source by default > > > > > > > > > > On Tue, May 21, 2013 at 6:53 PM, Shlomi [shlomi...@gmail.com](mailto:shlomi...@gmail.com) wrote: > > > > > > > > > > > Hey, > > > > > > > > > > > > We have some old java code that uses lucene and grizzly to serve > > > > > > queries over text. we have two field, a string field and a numeric (long) > > > > > > field. the indexing code is pretty straight forward. > > > > > > > > > > > > I was trying to migrate this to elastic, pretty simple configuration, > > > > > > and indexed the same data. > > > > > > > > > > > > the java based implementation took about 6gb, while to elastic took > > > > > > 17gb.. > > > > > > > > > > > > does this makes sense? what could i do about this? > > > > > > > > > > > > Thanks! > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > > > Groups "elasticsearch" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > send an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > > > > > > > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > > > > . > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Michael\_Sick](https://avatars.discourse-cdn.com/v4/letter/m/22d042/32.png) [@Michael\_Sick](https://discuss.elastic.co/u/Michael_Sick)\ **Post date:** [May 22, 2013, 1:13pm UTC](https://discuss.elastic.co/t/elasticsearch-index-much-larger-then-similar-lucene-index/12056/10 "2013-05-22T13:13:03Z")
Not sure about the storage - you might try [Google Code Archive - Long-term storage for Google Code Project Hosting.](https://code.google.com/p/luke/) and [GitHub - jprante/elasticsearch-skywalker: Skywalker for Elasticsearch is like Luke for Lucene](https://github.com/jprante/elasticsearch-skywalker#readme) to see into your indicies. I have not used either but had bookmarked for just such an occasion. On Wed, May 22, 2013 at 5:08 AM, Shlomi [shlomivaknin@gmail.com](mailto:shlomivaknin@gmail.com) wrote: > does ES store its numeric fields as strings? > > can someone confirm that if you disable \_source and keep each field as > stored and indexed, your fields becomes invisible (although queriable)? or > am i doing something totally wrong?.. > > Thanks > > On Tuesday, May 21, 2013 7:10:07 PM UTC+3, Shlomi wrote: > > > here is a fraction of the mapping i have (i use clojure so its a bit > > different from json, but its essentially the same): > > > > ``` > > {:test { > > :_source {:enabled "false" } > > :_all {:enabled "false" } > > :properties {:gram {:type "string" :store "yes" > > > > ``` > > > > :analyzer :ngram-index :compress "true"} > > :freq {:type "long" :store > > "yes"} }}}] > > > > On Tuesday, May 21, 2013 7:07:44 PM UTC+3, Shlomi wrote: > > > > > Hey, > > > > > > thanks all, let me reply: > > > > > > Michael - no, i set replicas to 0 (if that what you meant..) > > > > > > Itamar & Matt - i disabled \_all and \_source, and explicitly set "store" > > > to "yes" for both fields (i dont care about perf for now..) - with this > > > setting i still got a much larger size and was still unable to see the > > > fields (although i set store to yes) through queries (only got id's back) > > > > > > On Tuesday, May 21, 2013 7:03:19 PM UTC+3, Matt Weber wrote: > > > > > > > Don't forget about the \_all field. Also, if you don't store the > > > > source, you need to explicitly set "store" to yes on your field mappings so > > > > you can have them returned in the results. > > > > > > > > On Tue, May 21, 2013 at 8:59 AM, Shlomi [shlomi...@gmail.com](mailto:shlomi...@gmail.com) wrote: > > > > > > > > > yes, so i was trying to exclude source, but then queries didnt return > > > > > anything besides id. but in any case, even disabling source still gave me a > > > > > large index.. > > > > > > > > > > any way to tell it to save just the fields? > > > > > > > > > > On Tuesday, May 21, 2013 6:54:38 PM UTC+3, Itamar Syn-Hershko wrote: > > > > > > > > > > > Yes, because ES stores the entire source by default > > > > > > > > > > > > On Tue, May 21, 2013 at 6:53 PM, Shlomi [shlomi...@gmail.com](mailto:shlomi...@gmail.com) wrote: > > > > > > > > > > > > > Hey, > > > > > > > > > > > > > > We have some old java code that uses lucene and grizzly to serve > > > > > > > queries over text. we have two field, a string field and a numeric (long) > > > > > > > field. the indexing code is pretty straight forward. > > > > > > > > > > > > > > I was trying to migrate this to elastic, pretty simple > > > > > > > configuration, and indexed the same data. > > > > > > > > > > > > > > the java based implementation took about 6gb, while to elastic took > > > > > > > 17gb.. > > > > > > > > > > > > > > does this makes sense? what could i do about this? > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > -- > > > > > > > You received this message because you are subscribed to the Google > > > > > > > Groups "elasticsearch" group. > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > send an email to elasticsearc...@\*\*googlegroups.\*\*com. > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/\*\*grou\*\*ps/opt\_out](https://groups.google.com/ **grou** ps/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > > > > > . > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > > > Groups "elasticsearch" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > > an email to elasticsearc...@googlegroups.\*\*com. > > > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > > > > . > > > > > > > > -- > > > > You received this message because you are subscribed to the Google Groups > > > > "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send an > > > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![shlomivaknin](https://avatars.discourse-cdn.com/v4/letter/s/4da419/32.png) [@shlomivaknin](https://discuss.elastic.co/u/shlomivaknin)\ **Post date:** [May 22, 2013, 1:49pm UTC](https://discuss.elastic.co/t/elasticsearch-index-much-larger-then-similar-lucene-index/12056/11 "2013-05-22T13:49:32Z")
Hey Michael I managed to find my fields (had to manually ask for them). The question remains though, why is the database so big. ill give skywalker a chance, see maybe it will shed some light on this situation.. The weird thing is that even though i disabled \_source and \_all, the size remained the same... meaning 17gb instead of 7gb. thats a lot of wasted space... If anyone has any more ideas why elastic is so ridiculously large compared to a straight forward lucene, i am very interested to hear On Wednesday, May 22, 2013 4:13:03 PM UTC+3, Michael Sick wrote: > Not sure about the storage - you might try [Google Code Archive - Long-term storage for Google Code Project Hosting.](https://code.google.com/p/luke/) > and [GitHub - jprante/elasticsearch-skywalker: Skywalker for Elasticsearch is like Luke for Lucene](https://github.com/jprante/elasticsearch-skywalker#readme) to see > into your indicies. I have not used either but had bookmarked for just such > an occasion. > > On Wed, May 22, 2013 at 5:08 AM, Shlomi \<[shlomi...@gmail.com](mailto:shlomi...@gmail.com) \ > > > wrote: > > > does ES store its numeric fields as strings? > > > > can someone confirm that if you disable \_source and keep each field as > > stored and indexed, your fields becomes invisible (although queriable)? or > > am i doing something totally wrong?.. > > > > Thanks > > > > On Tuesday, May 21, 2013 7:10:07 PM UTC+3, Shlomi wrote: > > > > > here is a fraction of the mapping i have (i use clojure so its a bit > > > different from json, but its essentially the same): > > > > > > ``` > > > {:test { > > > :_source {:enabled "false" } > > > :_all {:enabled "false" } > > > :properties {:gram {:type "string" :store "yes" > > > > > > ``` > > > > > > :analyzer :ngram-index :compress "true"} > > > :freq {:type "long" :store > > > "yes"} }}}] > > > > > > On Tuesday, May 21, 2013 7:07:44 PM UTC+3, Shlomi wrote: > > > > > > > Hey, > > > > > > > > thanks all, let me reply: > > > > > > > > Michael - no, i set replicas to 0 (if that what you meant..) > > > > > > > > Itamar & Matt - i disabled \_all and \_source, and explicitly set "store" > > > > to "yes" for both fields (i dont care about perf for now..) - with this > > > > setting i still got a much larger size and was still unable to see the > > > > fields (although i set store to yes) through queries (only got id's back) > > > > > > > > On Tuesday, May 21, 2013 7:03:19 PM UTC+3, Matt Weber wrote: > > > > > > > > > Don't forget about the \_all field. Also, if you don't store the > > > > > source, you need to explicitly set "store" to yes on your field mappings so > > > > > you can have them returned in the results. > > > > > > > > > > On Tue, May 21, 2013 at 8:59 AM, Shlomi [shlomi...@gmail.com](mailto:shlomi...@gmail.com) wrote: > > > > > > > > > > > yes, so i was trying to exclude source, but then queries didnt return > > > > > > anything besides id. but in any case, even disabling source still gave me a > > > > > > large index.. > > > > > > > > > > > > any way to tell it to save just the fields? > > > > > > > > > > > > On Tuesday, May 21, 2013 6:54:38 PM UTC+3, Itamar Syn-Hershko wrote: > > > > > > > > > > > > > Yes, because ES stores the entire source by default > > > > > > > > > > > > > > On Tue, May 21, 2013 at 6:53 PM, Shlomi [shlomi...@gmail.com](mailto:shlomi...@gmail.com) wrote: > > > > > > > > > > > > > > > Hey, > > > > > > > > > > > > > > > > We have some old java code that uses lucene and grizzly to serve > > > > > > > > queries over text. we have two field, a string field and a numeric (long) > > > > > > > > field. the indexing code is pretty straight forward. > > > > > > > > > > > > > > > > I was trying to migrate this to elastic, pretty simple > > > > > > > > configuration, and indexed the same data. > > > > > > > > > > > > > > > > the java based implementation took about 6gb, while to elastic took > > > > > > > > 17gb.. > > > > > > > > > > > > > > > > does this makes sense? what could i do about this? > > > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > > > -- > > > > > > > > You received this message because you are subscribed to the Google > > > > > > > > Groups "elasticsearch" group. > > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > > send an email to elasticsearc...@\*\*googlegroups.\*\*com. > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/\*\*grou](https://groups.google.com/**grou)\*\* > > > > > > > > ps/opt\_out [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > > > > > > > -- > > > > > > > You received this message because you are subscribed to the Google > > > > > > > Groups "elasticsearch" group. > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > send an email to elasticsearc...@googlegroups.\*\*com. > > > > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > > > > > . > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google Groups > > > > > "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send an > > > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Mark\_Harwood1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood1/32/101255_2.png) [@Mark\_Harwood1](https://discuss.elastic.co/u/Mark_Harwood1)\ **Post date:** [May 22, 2013, 2:10pm UTC](https://discuss.elastic.co/t/elasticsearch-index-much-larger-then-similar-lucene-index/12056/12 "2013-05-22T14:10:31Z")
How many shards do you have? A multi-sharded ES index will cost you more than a single Lucene index due to duplication of index terms./worse postings compression if the same content is split across many lucene indexes. On Wednesday, May 22, 2013 2:49:32 PM UTC+1, Shlomi wrote: > Hey Michael > > I managed to find my fields (had to manually ask for them). The question > remains though, why is the database so big. ill give skywalker a chance, > see maybe it will shed some light on this situation.. > > The weird thing is that even though i disabled \_source and \_all, the size > remained the same... meaning 17gb instead of 7gb. thats a lot of wasted > space... > > If anyone has any more ideas why elastic is so ridiculously large compared > to a straight forward lucene, i am very interested to hear > > On Wednesday, May 22, 2013 4:13:03 PM UTC+3, Michael Sick wrote: > > > Not sure about the storage - you might try > > [Google Code Archive - Long-term storage for Google Code Project Hosting.](https://code.google.com/p/luke/) and > > [GitHub - jprante/elasticsearch-skywalker: Skywalker for Elasticsearch is like Luke for Lucene](https://github.com/jprante/elasticsearch-skywalker#readme) to see into > > your indicies. I have not used either but had bookmarked for just such an > > occasion. > > > > On Wed, May 22, 2013 at 5:08 AM, Shlomi [shlomi...@gmail.com](mailto:shlomi...@gmail.com) wrote: > > > > > does ES store its numeric fields as strings? > > > > > > can someone confirm that if you disable \_source and keep each field as > > > stored and indexed, your fields becomes invisible (although queriable)? or > > > am i doing something totally wrong?.. > > > > > > Thanks > > > > > > On Tuesday, May 21, 2013 7:10:07 PM UTC+3, Shlomi wrote: > > > > > > > here is a fraction of the mapping i have (i use clojure so its a bit > > > > different from json, but its essentially the same): > > > > > > > > ``` > > > > {:test { > > > > :_source {:enabled "false" } > > > > :_all {:enabled "false" } > > > > :properties {:gram {:type "string" :store "yes" > > > > > > > > ``` > > > > > > > > :analyzer :ngram-index :compress "true"} > > > > :freq {:type "long" :store > > > > "yes"} }}}] > > > > > > > > On Tuesday, May 21, 2013 7:07:44 PM UTC+3, Shlomi wrote: > > > > > > > > > Hey, > > > > > > > > > > thanks all, let me reply: > > > > > > > > > > Michael - no, i set replicas to 0 (if that what you meant..) > > > > > > > > > > Itamar & Matt - i disabled \_all and \_source, and explicitly set > > > > > "store" to "yes" for both fields (i dont care about perf for now..) - with > > > > > this setting i still got a much larger size and was still unable to see the > > > > > fields (although i set store to yes) through queries (only got id's back) > > > > > > > > > > On Tuesday, May 21, 2013 7:03:19 PM UTC+3, Matt Weber wrote: > > > > > > > > > > > Don't forget about the \_all field. Also, if you don't store the > > > > > > source, you need to explicitly set "store" to yes on your field mappings so > > > > > > you can have them returned in the results. > > > > > > > > > > > > On Tue, May 21, 2013 at 8:59 AM, Shlomi [shlomi...@gmail.com](mailto:shlomi...@gmail.com) wrote: > > > > > > > > > > > > > yes, so i was trying to exclude source, but then queries didnt > > > > > > > return anything besides id. but in any case, even disabling source still > > > > > > > gave me a large index.. > > > > > > > > > > > > > > any way to tell it to save just the fields? > > > > > > > > > > > > > > On Tuesday, May 21, 2013 6:54:38 PM UTC+3, Itamar Syn-Hershko wrote: > > > > > > > > > > > > > > > Yes, because ES stores the entire source by default > > > > > > > > > > > > > > > > On Tue, May 21, 2013 at 6:53 PM, Shlomi [shlomi...@gmail.com](mailto:shlomi...@gmail.com)wrote: > > > > > > > > > > > > > > > > > Hey, > > > > > > > > > > > > > > > > > > We have some old java code that uses lucene and grizzly to serve > > > > > > > > > queries over text. we have two field, a string field and a numeric (long) > > > > > > > > > field. the indexing code is pretty straight forward. > > > > > > > > > > > > > > > > > > I was trying to migrate this to elastic, pretty simple > > > > > > > > > configuration, and indexed the same data. > > > > > > > > > > > > > > > > > > the java based implementation took about 6gb, while to elastic > > > > > > > > > took 17gb.. > > > > > > > > > > > > > > > > > > does this makes sense? what could i do about this? > > > > > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > > > > > -- > > > > > > > > > You received this message because you are subscribed to the Google > > > > > > > > > Groups "elasticsearch" group. > > > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > > > send an email to elasticsearc...@\*\*googlegroups.\*\*com. > > > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/\*\*grou](https://groups.google.com/**grou)\*\* > > > > > > > > > ps/opt\_out [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > > > > > > > > > -- > > > > > > > > You received this message because you are subscribed to the Google > > > > > > > > Groups "elasticsearch" group. > > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > > send an email to elasticsearc...@googlegroups.\*\*com. > > > > > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > > > > > > . > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > > > Groups "elasticsearch" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![David\_G\_Ortega](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/david_g_ortega/32/46359_2.png) [@David\_G\_Ortega](https://discuss.elastic.co/u/David_G_Ortega)\ **Post date:** [May 22, 2013, 2:10pm UTC](https://discuss.elastic.co/t/elasticsearch-index-much-larger-then-similar-lucene-index/12056/13 "2013-05-22T14:10:37Z")
I had the same size issue but was exactly what the colleagues have pointed, \_all and \_source enabled, makes sense... they only thing I can think and its so silly that really ashames me to ask you is "have you deleted the index before apply the mapping?" I also see that you are using ngrams... I suppose that you use that in the vanilla lucene index... -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [May 22, 2013, 2:10pm UTC](https://discuss.elastic.co/t/elasticsearch-index-much-larger-then-similar-lucene-index/12056/14 "2013-05-22T14:10:41Z")
Please note, Skywalker needs an update for 0.90 - it is still on 0.20. The update is in progress. Jörg Am 22.05.13 15:49, schrieb Shlomi: > I managed to find my fields (had to manually ask for them). The > question remains though, why is the database so big. ill give > skywalker a chance, see maybe it will shed some light on this situation.. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [May 22, 2013, 2:22pm UTC](https://discuss.elastic.co/t/elasticsearch-index-much-larger-then-similar-lucene-index/12056/15 "2013-05-22T14:22:53Z")
You are using ngram tokenizer which explodes index size. If you use ES default sharding, you have 5 shards (and therefore, 5 Lucene indexes). With ngram, you have scattered tokens over all shards, and this converges to 5x the space compared to 1 shard. Also, store = yes for each field is kind of clumsy. You have to enable each field to get them returned for a query (only \_source is returned by default). I don't see much sense in making an ngram analyzed field stored. Can you elaborate? Jörg Am 22.05.13 11:08, schrieb Shlomi: > does ES store its numeric fields as strings? > > can someone confirm that if you disable \_source and keep each field as > stored and indexed, your fields becomes invisible (although > queriable)? or am i doing something totally wrong?.. > > Thanks > > On Tuesday, May 21, 2013 7:10:07 PM UTC+3, Shlomi wrote: > > ``` > here is a fraction of the mapping i have (i use clojure so its a > bit different from json, but its essentially the same): > > {:test { > :_source {:enabled "false" } > :_all {:enabled "false" } > :properties {:gram {:type "string" :store > "yes" :analyzer :ngram-index :compress "true"} > :freq {:type "long" > :store "yes"} }}}] > > On Tuesday, May 21, 2013 7:07:44 PM UTC+3, Shlomi wrote: > > Hey, > > thanks all, let me reply: > > Michael - no, i set replicas to 0 (if that what you meant..) > > Itamar & Matt - i disabled _all and _source, and explicitly > set "store" to "yes" for both fields (i dont care about perf > for now..) - with this setting i still got a much larger size > and was still unable to see the fields (although i set store > to yes) through queries (only got id's back) > > On Tuesday, May 21, 2013 7:03:19 PM UTC+3, Matt Weber wrote: > > Don't forget about the _all field. Also, if you don't > store the source, you need to explicitly set "store" to > yes on your field mappings so you can have them returned > in the results. > > On Tue, May 21, 2013 at 8:59 AM, Shlomi > wrote: > > yes, so i was trying to exclude source, but then > queries didnt return anything besides id. but in any > case, even disabling source still gave me a large index.. > > any way to tell it to save just the fields? > > On Tuesday, May 21, 2013 6:54:38 PM UTC+3, Itamar > Syn-Hershko wrote: > > Yes, because ES stores the entire source by default > > On Tue, May 21, 2013 at 6:53 PM, Shlomi > wrote: > > Hey, > > We have some old java code that uses lucene > and grizzly to serve queries over text. we > have two field, a string field and a numeric > (long) field. the indexing code is pretty > straight forward. > > I was trying to migrate this to elastic, > pretty simple configuration, and indexed the > same data. > > the java based implementation took about 6gb, > while to elastic took 17gb.. > > does this makes sense? what could i do about > this? > > Thanks! > > -- > You received this message because you are > subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop > receiving emails from it, send an email to > elasticsearc...@googlegroups.com. > > For more options, visit > https://groups.google.com/groups/opt_out > . > > -- > You received this message because you are subscribed > to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving > emails from it, send an email to > elasticsearc...@googlegroups.com. > For more options, visit > https://groups.google.com/groups/opt_out > . > > ``` > > -- > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![shlomivaknin](https://avatars.discourse-cdn.com/v4/letter/s/4da419/32.png) [@shlomivaknin](https://discuss.elastic.co/u/shlomivaknin)\ **Post date:** [May 22, 2013, 5:44pm UTC](https://discuss.elastic.co/t/elasticsearch-index-much-larger-then-similar-lucene-index/12056/16 "2013-05-22T17:44:22Z")
Hey, Thanks for replying, ngram is the name of the field, and is pre-computed: Jörg - I think i might have misled you, i am not using the ngram tokenizer, ":ngram-index" is a custom tokenizer that uses "lowercase" tokenizer, and a list of stopwords. David - Thanks for the suggestion, but yeah, my code fails if the index exists before it runs, this way i am sure the index was in fact deleted.. Mark - I tried with both a single shard and the default 5 shards. there was no different in size (surprisingly.. ) thanks for all your responses, but we have to keep thinking.. 🙂 On Wednesday, May 22, 2013 5:22:53 PM UTC+3, Jörg Prante wrote: > You are using ngram tokenizer which explodes index size. If you use ES > default sharding, you have 5 shards (and therefore, 5 Lucene indexes). > With ngram, you have scattered tokens over all shards, and this > converges to 5x the space compared to 1 shard. > > Also, store = yes for each field is kind of clumsy. You have to enable > each field to get them returned for a query (only \_source is returned by > default). I don't see much sense in making an ngram analyzed field > stored. Can you elaborate? > > Jörg > > Am 22.05.13 11:08, schrieb Shlomi: > > > does ES store its numeric fields as strings? > > > > can someone confirm that if you disable \_source and keep each field as > > stored and indexed, your fields becomes invisible (although > > queriable)? or am i doing something totally wrong?.. > > > > Thanks > > > > On Tuesday, May 21, 2013 7:10:07 PM UTC+3, Shlomi wrote: > > > > ``` > > here is a fraction of the mapping i have (i use clojure so its a > > bit different from json, but its essentially the same): > > > > {:test { > > :_source {:enabled "false" } > > :_all {:enabled "false" } > > :properties {:gram {:type "string" :store > > "yes" :analyzer :ngram-index :compress "true"} > > :freq {:type "long" > > :store "yes"} }}}] > > > > On Tuesday, May 21, 2013 7:07:44 PM UTC+3, Shlomi wrote: > > > > Hey, > > > > thanks all, let me reply: > > > > Michael - no, i set replicas to 0 (if that what you meant..) > > > > Itamar & Matt - i disabled _all and _source, and explicitly > > set "store" to "yes" for both fields (i dont care about perf > > for now..) - with this setting i still got a much larger size > > and was still unable to see the fields (although i set store > > to yes) through queries (only got id's back) > > > > On Tuesday, May 21, 2013 7:03:19 PM UTC+3, Matt Weber wrote: > > > > Don't forget about the _all field. Also, if you don't > > store the source, you need to explicitly set "store" to > > yes on your field mappings so you can have them returned > > in the results. > > > > On Tue, May 21, 2013 at 8:59 AM, Shlomi > > wrote: > > > > yes, so i was trying to exclude source, but then > > queries didnt return anything besides id. but in any > > case, even disabling source still gave me a large > > > > ``` > > index.. > > > ``` > > any way to tell it to save just the fields? > > > > On Tuesday, May 21, 2013 6:54:38 PM UTC+3, Itamar > > Syn-Hershko wrote: > > > > Yes, because ES stores the entire source by default > > > > On Tue, May 21, 2013 at 6:53 PM, Shlomi > > wrote: > > > > Hey, > > > > We have some old java code that uses lucene > > and grizzly to serve queries over text. we > > have two field, a string field and a numeric > > (long) field. the indexing code is pretty > > straight forward. > > > > I was trying to migrate this to elastic, > > pretty simple configuration, and indexed the > > same data. > > > > the java based implementation took about 6gb, > > while to elastic took 17gb.. > > > > does this makes sense? what could i do about > > this? > > > > Thanks! > > > > -- > > You received this message because you are > > subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop > > receiving emails from it, send an email to > > elasticsearc...@googlegroups.com. > > > > For more options, visit > > https://groups.google.com/groups/opt_out > > . > > > > -- > > You received this message because you are subscribed > > to the Google Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving > > emails from it, send an email to > > elasticsearc...@googlegroups.com. > > For more options, visit > > https://groups.google.com/groups/opt_out > > . > > > > ``` > > > > -- > > You received this message because you are subscribed to the Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![mattweber](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mattweber/32/44940_2.png) [@mattweber](https://discuss.elastic.co/u/mattweber)\ **Post date:** [May 22, 2013, 6:27pm UTC](https://discuss.elastic.co/t/elasticsearch-index-much-larger-then-similar-lucene-index/12056/17 "2013-05-22T18:27:05Z")
Really we are just shooting in the dark here because of lack of information: What version of ES? What version of lucene? What does your lucene index settings (tokenizer, analyzers, etc) look like? Have you configured an ES mapping identical to what you use in lucene? How are you measuring your index size? Have your tried indexing a single document in lucene and ES and comparing the resulting index size? Gist us your mapping (not the clojure version) , custom analyzer settings, index settings, etc and we might be able to figure this out for you. Thanks, Matt Weber On Wed, May 22, 2013 at 10:44 AM, Shlomi [shlomivaknin@gmail.com](mailto:shlomivaknin@gmail.com) wrote: > Hey, > > Thanks for replying, ngram is the name of the field, and is pre-computed: > > Jörg - I think i might have misled you, i am not using the ngram > tokenizer, ":ngram-index" is a custom tokenizer that uses "lowercase" > tokenizer, and a list of stopwords. > > David - Thanks for the suggestion, but yeah, my code fails if the index > exists before it runs, this way i am sure the index was in fact deleted.. > > Mark - I tried with both a single shard and the default 5 shards. there > was no different in size (surprisingly.. ) > > thanks for all your responses, but we have to keep thinking.. 🙂 > > On Wednesday, May 22, 2013 5:22:53 PM UTC+3, Jörg Prante wrote: > > > You are using ngram tokenizer which explodes index size. If you use ES > > default sharding, you have 5 shards (and therefore, 5 Lucene indexes). > > With ngram, you have scattered tokens over all shards, and this > > converges to 5x the space compared to 1 shard. > > > > Also, store = yes for each field is kind of clumsy. You have to enable > > each field to get them returned for a query (only \_source is returned by > > default). I don't see much sense in making an ngram analyzed field > > stored. Can you elaborate? > > > > Jörg > > > > Am 22.05.13 11:08, schrieb Shlomi: > > > > > does ES store its numeric fields as strings? > > > > > > can someone confirm that if you disable \_source and keep each field as > > > stored and indexed, your fields becomes invisible (although > > > queriable)? or am i doing something totally wrong?.. > > > > > > Thanks > > > > > > On Tuesday, May 21, 2013 7:10:07 PM UTC+3, Shlomi wrote: > > > > > > ``` > > > here is a fraction of the mapping i have (i use clojure so its a > > > bit different from json, but its essentially the same): > > > > > > {:test { > > > :_source {:enabled "false" } > > > :_all {:enabled "false" } > > > :properties {:gram {:type "string" :store > > > "yes" :analyzer :ngram-index :compress "true"} > > > :freq {:type "long" > > > :store "yes"} }}}] > > > > > > On Tuesday, May 21, 2013 7:07:44 PM UTC+3, Shlomi wrote: > > > > > > Hey, > > > > > > thanks all, let me reply: > > > > > > Michael - no, i set replicas to 0 (if that what you meant..) > > > > > > Itamar & Matt - i disabled _all and _source, and explicitly > > > set "store" to "yes" for both fields (i dont care about perf > > > for now..) - with this setting i still got a much larger size > > > and was still unable to see the fields (although i set store > > > to yes) through queries (only got id's back) > > > > > > On Tuesday, May 21, 2013 7:03:19 PM UTC+3, Matt Weber wrote: > > > > > > Don't forget about the _all field. Also, if you don't > > > store the source, you need to explicitly set "store" to > > > yes on your field mappings so you can have them returned > > > in the results. > > > > > > On Tue, May 21, 2013 at 8:59 AM, Shlomi > > > wrote: > > > > > > yes, so i was trying to exclude source, but then > > > queries didnt return anything besides id. but in any > > > case, even disabling source still gave me a large > > > > > > ``` > > > > index.. > > > > > ``` > > > any way to tell it to save just the fields? > > > > > > On Tuesday, May 21, 2013 6:54:38 PM UTC+3, Itamar > > > Syn-Hershko wrote: > > > > > > Yes, because ES stores the entire source by default > > > > > > On Tue, May 21, 2013 at 6:53 PM, Shlomi > > > wrote: > > > > > > Hey, > > > > > > We have some old java code that uses lucene > > > and grizzly to serve queries over text. we > > > have two field, a string field and a numeric > > > (long) field. the indexing code is pretty > > > straight forward. > > > > > > I was trying to migrate this to elastic, > > > pretty simple configuration, and indexed the > > > same data. > > > > > > the java based implementation took about 6gb, > > > while to elastic took 17gb.. > > > > > > does this makes sense? what could i do about > > > this? > > > > > > Thanks! > > > > > > -- > > > You received this message because you are > > > subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop > > > receiving emails from it, send an email to > > > elasticsearc...@googlegroups.**com. > > > > > > For more options, visit > > > https://groups.google.com/**groups/opt_out > > > >. > > > > > > ``` > > > > > ``` > > > -- > > > You received this message because you are subscribed > > > to the Google Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving > > > emails from it, send an email to > > > elasticsearc...@googlegroups.**com. > > > For more options, visit > > > https://groups.google.com/**groups/opt_out > > > >. > > > > > > ``` > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![simonw\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simonw_2/32/1130_2.png) [@simonw\_2](https://discuss.elastic.co/u/simonw_2)\ **Post date:** [May 22, 2013, 7:35pm UTC](https://discuss.elastic.co/t/elasticsearch-index-much-larger-then-similar-lucene-index/12056/18 "2013-05-22T19:35:08Z")
I suggest you provide your lucene FieldTypes and your mapping, run your indexing against lucene and a single shard no-replica Elasticsearch instance. Then optimize the index and provide the output of ls -al on the index directory. it would also be interesting what exactly is "much larger". simon On Wednesday, May 22, 2013 8:27:05 PM UTC+2, Matt Weber wrote: > Really we are just shooting in the dark here because of lack of > information: > > What version of ES? What version of lucene? What does your lucene index > settings (tokenizer, analyzers, etc) look like? Have you configured an ES > mapping identical to what you use in lucene? How are you measuring your > index size? Have your tried indexing a single document in lucene and ES > and comparing the resulting index size? > > Gist us your mapping (not the clojure version) , custom analyzer settings, > index settings, etc and we might be able to figure this out for you. > > Thanks, > Matt Weber > > On Wed, May 22, 2013 at 10:44 AM, Shlomi \<[shlomi...@gmail.com](mailto:shlomi...@gmail.com)\ > > > wrote: > > > Hey, > > > > Thanks for replying, ngram is the name of the field, and is pre-computed: > > > > Jörg - I think i might have misled you, i am not using the ngram > > tokenizer, ":ngram-index" is a custom tokenizer that uses "lowercase" > > tokenizer, and a list of stopwords. > > > > David - Thanks for the suggestion, but yeah, my code fails if the index > > exists before it runs, this way i am sure the index was in fact deleted.. > > > > Mark - I tried with both a single shard and the default 5 shards. there > > was no different in size (surprisingly.. ) > > > > thanks for all your responses, but we have to keep thinking.. 🙂 > > > > On Wednesday, May 22, 2013 5:22:53 PM UTC+3, Jörg Prante wrote: > > > > > You are using ngram tokenizer which explodes index size. If you use ES > > > default sharding, you have 5 shards (and therefore, 5 Lucene indexes). > > > With ngram, you have scattered tokens over all shards, and this > > > converges to 5x the space compared to 1 shard. > > > > > > Also, store = yes for each field is kind of clumsy. You have to enable > > > each field to get them returned for a query (only \_source is returned by > > > default). I don't see much sense in making an ngram analyzed field > > > stored. Can you elaborate? > > > > > > Jörg > > > > > > Am 22.05.13 11:08, schrieb Shlomi: > > > > > > > does ES store its numeric fields as strings? > > > > > > > > can someone confirm that if you disable \_source and keep each field as > > > > stored and indexed, your fields becomes invisible (although > > > > queriable)? or am i doing something totally wrong?.. > > > > > > > > Thanks > > > > > > > > On Tuesday, May 21, 2013 7:10:07 PM UTC+3, Shlomi wrote: > > > > > > > > ``` > > > > here is a fraction of the mapping i have (i use clojure so its a > > > > bit different from json, but its essentially the same): > > > > > > > > {:test { > > > > :_source {:enabled "false" } > > > > :_all {:enabled "false" } > > > > :properties {:gram {:type "string" :store > > > > "yes" :analyzer :ngram-index :compress "true"} > > > > :freq {:type "long" > > > > :store "yes"} }}}] > > > > > > > > On Tuesday, May 21, 2013 7:07:44 PM UTC+3, Shlomi wrote: > > > > > > > > Hey, > > > > > > > > thanks all, let me reply: > > > > > > > > Michael - no, i set replicas to 0 (if that what you meant..) > > > > > > > > Itamar & Matt - i disabled _all and _source, and explicitly > > > > set "store" to "yes" for both fields (i dont care about perf > > > > for now..) - with this setting i still got a much larger size > > > > and was still unable to see the fields (although i set store > > > > to yes) through queries (only got id's back) > > > > > > > > On Tuesday, May 21, 2013 7:03:19 PM UTC+3, Matt Weber wrote: > > > > > > > > Don't forget about the _all field. Also, if you don't > > > > store the source, you need to explicitly set "store" to > > > > yes on your field mappings so you can have them returned > > > > in the results. > > > > > > > > On Tue, May 21, 2013 at 8:59 AM, Shlomi > > > > wrote: > > > > > > > > yes, so i was trying to exclude source, but then > > > > queries didnt return anything besides id. but in any > > > > case, even disabling source still gave me a large > > > > > > > > ``` > > > > > > index.. > > > > > > > ``` > > > > any way to tell it to save just the fields? > > > > > > > > On Tuesday, May 21, 2013 6:54:38 PM UTC+3, Itamar > > > > Syn-Hershko wrote: > > > > > > > > Yes, because ES stores the entire source by > > > > > > > > ``` > > > > > > default > > > > > > > ``` > > > > On Tue, May 21, 2013 at 6:53 PM, Shlomi > > > > wrote: > > > > > > > > Hey, > > > > > > > > We have some old java code that uses lucene > > > > and grizzly to serve queries over text. we > > > > have two field, a string field and a numeric > > > > (long) field. the indexing code is pretty > > > > straight forward. > > > > > > > > I was trying to migrate this to elastic, > > > > pretty simple configuration, and indexed the > > > > same data. > > > > > > > > the java based implementation took about 6gb, > > > > while to elastic took 17gb.. > > > > > > > > does this makes sense? what could i do about > > > > this? > > > > > > > > Thanks! > > > > > > > > -- > > > > You received this message because you are > > > > subscribed to the Google Groups > > > > "elasticsearch" group. > > > > To unsubscribe from this group and stop > > > > receiving emails from it, send an email to > > > > elasticsearc...@googlegroups.**com. > > > > > > > > For more options, visit > > > > https://groups.google.com/**groups/opt_out > > > > >. > > > > > > > > ``` > > > > > > > ``` > > > > -- > > > > You received this message because you are subscribed > > > > to the Google Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving > > > > emails from it, send an email to > > > > elasticsearc...@googlegroups.**com. > > > > For more options, visit > > > > https://groups.google.com/**groups/opt_out > > > > >. > > > > > > > > ``` > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [May 23, 2013, 2:34pm UTC](https://discuss.elastic.co/t/elasticsearch-index-much-larger-then-similar-lucene-index/12056/19 "2013-05-23T14:34:38Z")
Just wanted to add that I always encountered the same issue with Elasticsearch. Indices are almost twice as big despite aggressive trimming. I have simply come to accept the issue as a fact and moved on. 🙂 -- Ivan On Wed, May 22, 2013 at 12:35 PM, simonw [simon.willnauer@elasticsearch.com](mailto:simon.willnauer@elasticsearch.com)wrote: > I suggest you provide your lucene FieldTypes and your mapping, run your > indexing against lucene and a single shard no-replica Elasticsearch > instance. Then optimize the index and provide the output of ls -al on the > index directory. it would also be interesting what exactly is "much > larger". > > simon > > On Wednesday, May 22, 2013 8:27:05 PM UTC+2, Matt Weber wrote: > > > Really we are just shooting in the dark here because of lack of > > information: > > > > What version of ES? What version of lucene? What does your lucene index > > settings (tokenizer, analyzers, etc) look like? Have you configured an ES > > mapping identical to what you use in lucene? How are you measuring your > > index size? Have your tried indexing a single document in lucene and ES > > and comparing the resulting index size? > > > > Gist us your mapping (not the clojure version) , custom analyzer > > settings, index settings, etc and we might be able to figure this out for > > you. > > > > Thanks, > > Matt Weber > > > > On Wed, May 22, 2013 at 10:44 AM, Shlomi [shlomi...@gmail.com](mailto:shlomi...@gmail.com) wrote: > > > > > Hey, > > > > > > Thanks for replying, ngram is the name of the field, and is pre-computed: > > > > > > Jörg - I think i might have misled you, i am not using the ngram > > > tokenizer, ":ngram-index" is a custom tokenizer that uses "lowercase" > > > tokenizer, and a list of stopwords. > > > > > > David - Thanks for the suggestion, but yeah, my code fails if the index > > > exists before it runs, this way i am sure the index was in fact deleted.. > > > > > > Mark - I tried with both a single shard and the default 5 shards. there > > > was no different in size (surprisingly.. ) > > > > > > thanks for all your responses, but we have to keep thinking.. 🙂 > > > > > > On Wednesday, May 22, 2013 5:22:53 PM UTC+3, Jörg Prante wrote: > > > > > > > You are using ngram tokenizer which explodes index size. If you use ES > > > > default sharding, you have 5 shards (and therefore, 5 Lucene indexes). > > > > With ngram, you have scattered tokens over all shards, and this > > > > converges to 5x the space compared to 1 shard. > > > > > > > > Also, store = yes for each field is kind of clumsy. You have to enable > > > > each field to get them returned for a query (only \_source is returned > > > > by > > > > default). I don't see much sense in making an ngram analyzed field > > > > stored. Can you elaborate? > > > > > > > > Jörg > > > > > > > > Am 22.05.13 11:08, schrieb Shlomi: > > > > > > > > > does ES store its numeric fields as strings? > > > > > > > > > > can someone confirm that if you disable \_source and keep each field > > > > > as > > > > > stored and indexed, your fields becomes invisible (although > > > > > queriable)? or am i doing something totally wrong?.. > > > > > > > > > > Thanks > > > > > > > > > > On Tuesday, May 21, 2013 7:10:07 PM UTC+3, Shlomi wrote: > > > > > > > > > > ``` > > > > > here is a fraction of the mapping i have (i use clojure so its a > > > > > bit different from json, but its essentially the same): > > > > > > > > > > {:test { > > > > > :_source {:enabled "false" } > > > > > :_all {:enabled "false" } > > > > > :properties {:gram {:type "string" :store > > > > > "yes" :analyzer :ngram-index :compress "true"} > > > > > :freq {:type "long" > > > > > :store "yes"} }}}] > > > > > > > > > > On Tuesday, May 21, 2013 7:07:44 PM UTC+3, Shlomi wrote: > > > > > > > > > > Hey, > > > > > > > > > > thanks all, let me reply: > > > > > > > > > > Michael - no, i set replicas to 0 (if that what you meant..) > > > > > > > > > > Itamar & Matt - i disabled _all and _source, and explicitly > > > > > set "store" to "yes" for both fields (i dont care about perf > > > > > for now..) - with this setting i still got a much larger size > > > > > and was still unable to see the fields (although i set store > > > > > to yes) through queries (only got id's back) > > > > > > > > > > On Tuesday, May 21, 2013 7:03:19 PM UTC+3, Matt Weber wrote: > > > > > > > > > > Don't forget about the _all field. Also, if you don't > > > > > store the source, you need to explicitly set "store" to > > > > > yes on your field mappings so you can have them returned > > > > > in the results. > > > > > > > > > > On Tue, May 21, 2013 at 8:59 AM, Shlomi > > > > > wrote: > > > > > > > > > > yes, so i was trying to exclude source, but then > > > > > queries didnt return anything besides id. but in any > > > > > case, even disabling source still gave me a large > > > > > > > > > > ``` > > > > > > > > index.. > > > > > > > > > ``` > > > > > any way to tell it to save just the fields? > > > > > > > > > > On Tuesday, May 21, 2013 6:54:38 PM UTC+3, Itamar > > > > > Syn-Hershko wrote: > > > > > > > > > > Yes, because ES stores the entire source by > > > > > > > > > > ``` > > > > > > > > default > > > > > > > > > ``` > > > > > On Tue, May 21, 2013 at 6:53 PM, Shlomi > > > > > wrote: > > > > > > > > > > Hey, > > > > > > > > > > We have some old java code that uses lucene > > > > > and grizzly to serve queries over text. we > > > > > have two field, a string field and a numeric > > > > > (long) field. the indexing code is pretty > > > > > straight forward. > > > > > > > > > > I was trying to migrate this to elastic, > > > > > pretty simple configuration, and indexed the > > > > > same data. > > > > > > > > > > the java based implementation took about 6gb, > > > > > while to elastic took 17gb.. > > > > > > > > > > does this makes sense? what could i do about > > > > > this? > > > > > > > > > > Thanks! > > > > > > > > > > -- > > > > > You received this message because you are > > > > > subscribed to the Google Groups > > > > > "elasticsearch" group. > > > > > To unsubscribe from this group and stop > > > > > receiving emails from it, send an email to > > > > > elasticsearc...@googlegroups. **c** om. > > > > > > > > > > For more options, visit > > > > > https://groups.google.com/ **grou** ps/opt_out > > > > > >. > > > > > > > > > > ``` > > > > > > > > > ``` > > > > > -- > > > > > You received this message because you are subscribed > > > > > to the Google Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving > > > > > emails from it, send an email to > > > > > elasticsearc...@googlegroups. **c** om. > > > > > For more options, visit > > > > > https://groups.google.com/ **grou** ps/opt_out > > > > > >. > > > > > > > > > > ``` > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > send > > > > > an email to elasticsearc...@\*\*googlegroups.\*\*com. > > > > > For more options, visit [https://groups.google.com/\*\*grou\*\*ps/opt\_out](https://groups.google.com/ **grou** ps/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > > > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > > . > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Jerome\_Gagnon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerome_gagnon/32/2178_2.png) [@Jerome\_Gagnon](https://discuss.elastic.co/u/Jerome_Gagnon)\ **Post date:** [May 23, 2013, 2:54pm UTC](https://discuss.elastic.co/t/elasticsearch-index-much-larger-then-similar-lucene-index/12056/20 "2013-05-23T14:54:05Z")
+1 on that, we couldn't do much about it, we just hope that this doesn't affect the disk IO performance... On Thursday, May 23, 2013 10:34:38 AM UTC-4, Ivan Brusic wrote: > Just wanted to add that I always encountered the same issue with > Elasticsearch. Indices are almost twice as big despite aggressive trimming. > I have simply come to accept the issue as a fact and moved on. 🙂 > > -- > Ivan > > On Wed, May 22, 2013 at 12:35 PM, simonw \<[simon.w...@elasticsearch.com](mailto:simon.w...@elasticsearch.com)\ > > > wrote: > > > I suggest you provide your lucene FieldTypes and your mapping, run your > > indexing against lucene and a single shard no-replica Elasticsearch > > instance. Then optimize the index and provide the output of ls -al on the > > index directory. it would also be interesting what exactly is "much > > larger". > > > > simon > > > > On Wednesday, May 22, 2013 8:27:05 PM UTC+2, Matt Weber wrote: > > > > > Really we are just shooting in the dark here because of lack of > > > information: > > > > > > What version of ES? What version of lucene? What does your lucene > > > index settings (tokenizer, analyzers, etc) look like? Have you configured > > > an ES mapping identical to what you use in lucene? How are you measuring > > > your index size? Have your tried indexing a single document in lucene and > > > ES and comparing the resulting index size? > > > > > > Gist us your mapping (not the clojure version) , custom analyzer > > > settings, index settings, etc and we might be able to figure this out for > > > you. > > > > > > Thanks, > > > Matt Weber > > > > > > On Wed, May 22, 2013 at 10:44 AM, Shlomi [shlomi...@gmail.com](mailto:shlomi...@gmail.com) wrote: > > > > > > > Hey, > > > > > > > > Thanks for replying, ngram is the name of the field, and is > > > > pre-computed: > > > > > > > > Jörg - I think i might have misled you, i am not using the ngram > > > > tokenizer, ":ngram-index" is a custom tokenizer that uses "lowercase" > > > > tokenizer, and a list of stopwords. > > > > > > > > David - Thanks for the suggestion, but yeah, my code fails if the index > > > > exists before it runs, this way i am sure the index was in fact deleted.. > > > > > > > > Mark - I tried with both a single shard and the default 5 shards. there > > > > was no different in size (surprisingly.. ) > > > > > > > > thanks for all your responses, but we have to keep thinking.. 🙂 > > > > > > > > On Wednesday, May 22, 2013 5:22:53 PM UTC+3, Jörg Prante wrote: > > > > > > > > > You are using ngram tokenizer which explodes index size. If you use ES > > > > > default sharding, you have 5 shards (and therefore, 5 Lucene indexes). > > > > > With ngram, you have scattered tokens over all shards, and this > > > > > converges to 5x the space compared to 1 shard. > > > > > > > > > > Also, store = yes for each field is kind of clumsy. You have to enable > > > > > each field to get them returned for a query (only \_source is returned > > > > > by > > > > > default). I don't see much sense in making an ngram analyzed field > > > > > stored. Can you elaborate? > > > > > > > > > > Jörg > > > > > > > > > > Am 22.05.13 11:08, schrieb Shlomi: > > > > > > > > > > > does ES store its numeric fields as strings? > > > > > > > > > > > > can someone confirm that if you disable \_source and keep each field > > > > > > as > > > > > > stored and indexed, your fields becomes invisible (although > > > > > > queriable)? or am i doing something totally wrong?.. > > > > > > > > > > > > Thanks > > > > > > > > > > > > On Tuesday, May 21, 2013 7:10:07 PM UTC+3, Shlomi wrote: > > > > > > > > > > > > ``` > > > > > > here is a fraction of the mapping i have (i use clojure so its a > > > > > > bit different from json, but its essentially the same): > > > > > > > > > > > > {:test { > > > > > > :_source {:enabled "false" } > > > > > > :_all {:enabled "false" } > > > > > > :properties {:gram {:type "string" :store > > > > > > "yes" :analyzer :ngram-index :compress "true"} > > > > > > :freq {:type "long" > > > > > > :store "yes"} }}}] > > > > > > > > > > > > On Tuesday, May 21, 2013 7:07:44 PM UTC+3, Shlomi wrote: > > > > > > > > > > > > Hey, > > > > > > > > > > > > thanks all, let me reply: > > > > > > > > > > > > Michael - no, i set replicas to 0 (if that what you meant..) > > > > > > > > > > > > Itamar & Matt - i disabled _all and _source, and explicitly > > > > > > set "store" to "yes" for both fields (i dont care about perf > > > > > > for now..) - with this setting i still got a much larger > > > > > > > > > > > > ``` > > > > > > > > > > size > > > > > > > > > > > ``` > > > > > > and was still unable to see the fields (although i set store > > > > > > to yes) through queries (only got id's back) > > > > > > > > > > > > On Tuesday, May 21, 2013 7:03:19 PM UTC+3, Matt Weber wrote: > > > > > > > > > > > > Don't forget about the _all field. Also, if you don't > > > > > > store the source, you need to explicitly set "store" to > > > > > > yes on your field mappings so you can have them returned > > > > > > in the results. > > > > > > > > > > > > On Tue, May 21, 2013 at 8:59 AM, Shlomi > > > > > > wrote: > > > > > > > > > > > > yes, so i was trying to exclude source, but then > > > > > > queries didnt return anything besides id. but in any > > > > > > case, even disabling source still gave me a large > > > > > > > > > > > > ``` > > > > > > > > > > index.. > > > > > > > > > > > ``` > > > > > > any way to tell it to save just the fields? > > > > > > > > > > > > On Tuesday, May 21, 2013 6:54:38 PM UTC+3, Itamar > > > > > > Syn-Hershko wrote: > > > > > > > > > > > > Yes, because ES stores the entire source by > > > > > > > > > > > > ``` > > > > > > > > > > default > > > > > > > > > > > ``` > > > > > > On Tue, May 21, 2013 at 6:53 PM, Shlomi > > > > > > wrote: > > > > > > > > > > > > Hey, > > > > > > > > > > > > We have some old java code that uses lucene > > > > > > and grizzly to serve queries over text. we > > > > > > have two field, a string field and a numeric > > > > > > (long) field. the indexing code is pretty > > > > > > straight forward. > > > > > > > > > > > > I was trying to migrate this to elastic, > > > > > > pretty simple configuration, and indexed the > > > > > > same data. > > > > > > > > > > > > the java based implementation took about > > > > > > > > > > > > ``` > > > > > > > > > > 6gb, > > > > > > > > > > > ``` > > > > > > while to elastic took 17gb.. > > > > > > > > > > > > does this makes sense? what could i do about > > > > > > this? > > > > > > > > > > > > Thanks! > > > > > > > > > > > > -- > > > > > > You received this message because you are > > > > > > subscribed to the Google Groups > > > > > > "elasticsearch" group. > > > > > > To unsubscribe from this group and stop > > > > > > receiving emails from it, send an email to > > > > > > elasticsearc...@googlegroups. **c** om. > > > > > > > > > > > > For more options, visit > > > > > > https://groups.google.com/ **grou** ps/opt_out > > > > > > > > > > > > > > > > > ``` > > > > > > > > > > ups/opt\_out [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)\>. > > > > > > > > > > > ``` > > > > > > -- > > > > > > You received this message because you are subscribed > > > > > > to the Google Groups "elasticsearch" group. > > > > > > To unsubscribe from this group and stop receiving > > > > > > emails from it, send an email to > > > > > > elasticsearc...@googlegroups. **c** om. > > > > > > For more options, visit > > > > > > https://groups.google.com/ **grou** ps/opt_out > > > > > > >. > > > > > > > > > > > > ``` > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > > > Groups "elasticsearch" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > send > > > > > > an email to elasticsearc...@\*\*googlegroups.\*\*com. > > > > > > For more options, visit [https://groups.google.com/\*\*grou\*\*ps/opt\_out](https://groups.google.com/ **grou** ps/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > > > > > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > > > . > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). [Next page](https://discuss.elastic.co/t/elasticsearch-index-much-larger-then-similar-lucene-index/12056.md?page=2)