# Dynamic mapping with \*lots\* of fields? **URL:** \ **Category:** Elasticsearch\ **Created:** [April 27, 2014, 11:15am UTC](https://discuss.elastic.co/t/dynamic-mapping-with-lots-of-fields/17220 "2014-04-27T11:15:01Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Daniel\_Winterstein](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daniel_winterstein/32/1442_2.png) [@Daniel\_Winterstein](https://discuss.elastic.co/u/Daniel_Winterstein)\ **Post date:** [April 27, 2014, 11:15am UTC](https://discuss.elastic.co/t/dynamic-mapping-with-lots-of-fields/17220/1 "2014-04-27T11:15:01Z")
Hello, I have a case where each document is likely to introduce some specific & often unique properties -- leading to millions or more fields. The values stored under these should be searchable. But the mapping structure itself is not used in queries. Having ElasticSearch treat these as dynamic fields works -- but how well does ElasticSearch cope with very large mappings? Am I right in thinking each property adds a field to the mapping, which is shared & held in memory by all nodes? So a mapping can become too large, and create performance issues. E.g 100 million documents might lead to gigabytes of ram locked up modelling the mapping. Is that correct? You can switch off dynamic fields altogether -- but then I think the content stored within them becomes unsearchable. Is there a way to have dynamic fields accessible to search, without modelling them as part of the mapping? Thank you & Best regards, - Daniel -- Dr Daniel Winterstein Director A: CodeBase Argyle House, Edinburgh, EH3 9DR M: +44 (0)772 5172 612 [http://winterwell.com](http://winterwell.com)[http://sodash.com](http://sodash.com) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEmLStmrGs%2BHaW%3De1XChi%3DFQ\_LKEPEntRUvT8zt\_BJKPUESkeg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEmLStmrGs%2BHaW%3De1XChi%3DFQ_LKEPEntRUvT8zt_BJKPUESkeg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [April 28, 2014, 7:13am UTC](https://discuss.elastic.co/t/dynamic-mapping-with-lots-of-fields/17220/2 "2014-04-28T07:13:17Z")
What is your data model? It seems you could live with search on a single field only. Then I conclude why not use key/value stream, search on value, and use highlighting for looking up the key(s) in the result doc? { "key" : "...", "value" : "..." } Jörg On Sun, Apr 27, 2014 at 1:15 PM, Daniel Winterstein \< [daniel.winterstein@gmail.com](mailto:daniel.winterstein@gmail.com)\> wrote: > Hello, > > I have a case where each document is likely to introduce some specific > & often unique properties -- leading to millions or more fields. > The values stored under these should be searchable. But the mapping > structure itself is not used in queries. > > Having Elasticsearch treat these as dynamic fields works -- but how > well does Elasticsearch cope with very large mappings? > > Am I right in thinking each property adds a field to the mapping, > which is shared & held in memory by all nodes? > So a mapping can become too large, and create performance issues. E.g > 100 million documents might lead to gigabytes of ram locked up > modelling the mapping. > Is that correct? > > You can switch off dynamic fields altogether -- but then I think the > content stored within them becomes unsearchable. > Is there a way to have dynamic fields accessible to search, without > modelling them as part of the mapping? > > Thank you & > Best regards, > > - Daniel > > -- > Dr Daniel Winterstein > Director > > A: CodeBase Argyle House, Edinburgh, EH3 9DR > M: +44 (0)772 5172 612 > [http://winterwell.com](http://winterwell.com)[http://sodash.com](http://sodash.com) > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAEmLStmrGs%2BHaW%3De1XChi%3DFQ\_LKEPEntRUvT8zt\_BJKPUESkeg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEmLStmrGs%2BHaW%3De1XChi%3DFQ_LKEPEntRUvT8zt_BJKPUESkeg%40mail.gmail.com) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHZfdBS31PB4aOMgusfrND1poGmDbb-%2B9So28x1ZDeKFQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHZfdBS31PB4aOMgusfrND1poGmDbb-%2B9So28x1ZDeKFQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Daniel\_Winterstein](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daniel_winterstein/32/1442_2.png) [@Daniel\_Winterstein](https://discuss.elastic.co/u/Daniel_Winterstein)\ **Post date:** [April 28, 2014, 2:42pm UTC](https://discuss.elastic.co/t/dynamic-mapping-with-lots-of-fields/17220/3 "2014-04-28T14:42:42Z")
Hello Jörg, > What is your data model? > It seems you could live with search on a single field only. A relevant point I left out: I have concurrent non-conflicting edits going on, which I want to merge. I.e. several users setting their own dynamic fields at the same time. Using separate fields with the update command makes this painless. NB: I'm assuming ES handles concurrent updates nicely, but I haven't stress-tested this yet. If we had a single-field, we'd need to introduce versioning and conflict handling. > Then I conclude why not use key/value stream, search on value, and use > highlighting for looking up the key(s) in the result doc? I'm not sure I understand what you're proposing here. NB: The alternatives I'm considering are: 1. Lots of dynamic fields -- with the concerns talked about in this thread. 2. In my code, splitting edits to the "concurrent fields" off from other edits. Then using normal put and update for normal fields, and MVEL update scripts for edits to concurrent fields. 3. Versioning and conflict-handling. 4. Writing a plugin for the ES backend to allow for custom merge logic on a single field. Best regards, - Daniel On Monday, 28 April 2014 08:13:17 UTC+1, Jörg Prante wrote: > What is your data model? > > It seems you could live with search on a single field only. > > Then I conclude why not use key/value stream, search on value, and use > highlighting for looking up the key(s) in the result doc? > > { > "key" : "...", > "value" : "..." > } > > Jörg > > On Sun, Apr 27, 2014 at 1:15 PM, Daniel Winterstein \< > [daniel.wi...@gmail.com](mailto:daniel.wi...@gmail.com) \\> wrote: > > > Hello, > > > > I have a case where each document is likely to introduce some specific > > & often unique properties -- leading to millions or more fields. > > The values stored under these should be searchable. But the mapping > > structure itself is not used in queries. > > > > Having Elasticsearch treat these as dynamic fields works -- but how > > well does Elasticsearch cope with very large mappings? > > > > Am I right in thinking each property adds a field to the mapping, > > which is shared & held in memory by all nodes? > > So a mapping can become too large, and create performance issues. E.g > > 100 million documents might lead to gigabytes of ram locked up > > modelling the mapping. > > Is that correct? > > > > You can switch off dynamic fields altogether -- but then I think the > > content stored within them becomes unsearchable. > > Is there a way to have dynamic fields accessible to search, without > > modelling them as part of the mapping? > > > > Thank you & > > Best regards, > > > > - Daniel > > > > -- > > Dr Daniel Winterstein > > Director > > > > A: CodeBase Argyle House, Edinburgh, EH3 9DR > > M: +44 (0)772 5172 612 > > [http://winterwell.com](http://winterwell.com)[http://sodash.com](http://sodash.com) > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAEmLStmrGs%2BHaW%3De1XChi%3DFQ\_LKEPEntRUvT8zt\_BJKPUESkeg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEmLStmrGs%2BHaW%3De1XChi%3DFQ_LKEPEntRUvT8zt_BJKPUESkeg%40mail.gmail.com) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/0f402257-4d5a-4b69-b0a3-4cc5e939aaa8%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0f402257-4d5a-4b69-b0a3-4cc5e939aaa8%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:33am UTC](https://discuss.elastic.co/t/dynamic-mapping-with-lots-of-fields/17220/4 "2017-07-06T01:33:04Z")