# Json documents with large number of fields **URL:** \ **Category:** Elasticsearch\ **Created:** [November 23, 2014, 9:19am UTC](https://discuss.elastic.co/t/json-documents-with-large-number-of-fields/20903 "2014-11-23T09:19:50Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Ajay\_Divakaran](https://avatars.discourse-cdn.com/v4/letter/a/779978/32.png) [@Ajay\_Divakaran](https://discuss.elastic.co/u/Ajay_Divakaran)\ **Post date:** [November 23, 2014, 9:19am UTC](https://discuss.elastic.co/t/json-documents-with-large-number-of-fields/20903/1 "2014-11-23T09:19:50Z")
Hi all, I have around 20K JSON documents that have around 350 fields to be pushed into ES (0.90.13). The mapping type is all multi valued string fields. Currently I have a single shard and 0 replicas. My question is whether ES(Lucene) is capable of handling these large documents? Or does it work well for small documents (less fields) but larger number of such small documents? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1d746894-2e54-4215-a489-bda5708515f1%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1d746894-2e54-4215-a489-bda5708515f1%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [November 23, 2014, 2:13pm UTC](https://discuss.elastic.co/t/json-documents-with-large-number-of-fields/20903/2 "2014-11-23T14:13:20Z")
This is not large. I have 5500 fields in ~100m docs, most fields are not analyzed / not indexed. Sure, this works perfectly. Jörg On Sun, Nov 23, 2014 at 10:19 AM, Ajay Divakaran \<[ajay.divakaran86@gmail.com](mailto:ajay.divakaran86@gmail.com) > wrote: > Hi all, > > I have around 20K JSON documents that have around 350 fields to be pushed > into ES (0.90.13). The mapping type is all multi valued string fields. > Currently I have a single shard and 0 replicas. > > My question is whether ES(Lucene) is capable of handling these large > documents? Or does it work well for small documents (less fields) but > larger number of such small documents? > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/1d746894-2e54-4215-a489-bda5708515f1%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1d746894-2e54-4215-a489-bda5708515f1%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/1d746894-2e54-4215-a489-bda5708515f1%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/1d746894-2e54-4215-a489-bda5708515f1%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGoO1jaUYV%2BM3adsFA%2BTfH5yXRG1r7ZMMTsKsdW%3DwN9NQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGoO1jaUYV%2BM3adsFA%2BTfH5yXRG1r7ZMMTsKsdW%3DwN9NQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Ajay\_Divakaran](https://avatars.discourse-cdn.com/v4/letter/a/779978/32.png) [@Ajay\_Divakaran](https://discuss.elastic.co/u/Ajay_Divakaran)\ **Post date:** [November 23, 2014, 5:22pm UTC](https://discuss.elastic.co/t/json-documents-with-large-number-of-fields/20903/3 "2014-11-23T17:22:35Z")
Jorg, Thanks for your reply. Does the memory spike when you retrieve these documents in bulk as a result of a query/filter result? On Sunday, November 23, 2014 7:43:49 PM UTC+5:30, Jörg Prante wrote: > This is not large. I have 5500 fields in ~100m docs, most fields are not > analyzed / not indexed. Sure, this works perfectly. > > Jörg > > On Sun, Nov 23, 2014 at 10:19 AM, Ajay Divakaran \<[ajay.div...@gmail.com](mailto:ajay.div...@gmail.com) > \\> wrote: > > > Hi all, > > > > I have around 20K JSON documents that have around 350 fields to be pushed > > into ES (0.90.13). The mapping type is all multi valued string fields. > > Currently I have a single shard and 0 replicas. > > > > My question is whether ES(Lucene) is capable of handling these large > > documents? Or does it work well for small documents (less fields) but > > larger number of such small documents? > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/1d746894-2e54-4215-a489-bda5708515f1%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1d746894-2e54-4215-a489-bda5708515f1%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/1d746894-2e54-4215-a489-bda5708515f1%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/1d746894-2e54-4215-a489-bda5708515f1%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/00b46a98-4d64-40d9-a011-b8b7d29dcaa6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/00b46a98-4d64-40d9-a011-b8b7d29dcaa6%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [November 23, 2014, 10:37pm UTC](https://discuss.elastic.co/t/json-documents-with-large-number-of-fields/20903/4 "2014-11-23T22:37:03Z")
No. Use filtered query to avoid memory spikes by post filter. Jörg On Sun, Nov 23, 2014 at 6:22 PM, Ajay Divakaran [ajay.divakaran86@gmail.com](mailto:ajay.divakaran86@gmail.com) wrote: > Jorg, > > Thanks for your reply. > > Does the memory spike when you retrieve these documents in bulk as a > result of a query/filter result? > > On Sunday, November 23, 2014 7:43:49 PM UTC+5:30, Jörg Prante wrote: > > > This is not large. I have 5500 fields in ~100m docs, most fields are not > > analyzed / not indexed. Sure, this works perfectly. > > > > Jörg > > > > On Sun, Nov 23, 2014 at 10:19 AM, Ajay Divakaran [ajay.div...@gmail.com](mailto:ajay.div...@gmail.com) > > wrote: > > > > > Hi all, > > > > > > I have around 20K JSON documents that have around 350 fields to be > > > pushed into ES (0.90.13). The mapping type is all multi valued string > > > fields. > > > Currently I have a single shard and 0 replicas. > > > > > > My question is whether ES(Lucene) is capable of handling these large > > > documents? Or does it work well for small documents (less fields) but > > > larger number of such small documents? > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/1d746894-2e54-4215-a489-bda5708515f1% > > > [40googlegroups.com](http://40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/1d746894-2e54-4215-a489-bda5708515f1%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/1d746894-2e54-4215-a489-bda5708515f1%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/00b46a98-4d64-40d9-a011-b8b7d29dcaa6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/00b46a98-4d64-40d9-a011-b8b7d29dcaa6%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/00b46a98-4d64-40d9-a011-b8b7d29dcaa6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/00b46a98-4d64-40d9-a011-b8b7d29dcaa6%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE9cLxhNgf0-PX4xBZ%3D\_ZPx8%3DkFSumhCcW9y%2Bx\_kJgSyA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE9cLxhNgf0-PX4xBZ%3D_ZPx8%3DkFSumhCcW9y%2Bx_kJgSyA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:48am UTC](https://discuss.elastic.co/t/json-documents-with-large-number-of-fields/20903/5 "2017-07-06T00:48:11Z")