# Structure elasticsearch for 50 million fields **URL:** \ **Category:** Elasticsearch\ **Created:** [June 4, 2019, 9:40am UTC](https://discuss.elastic.co/t/structure-elasticsearch-for-50-million-fields/184134 "2019-06-04T09:40:53Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![Niels\_van\_Rijn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niels_van_rijn/32/45639_2.png) [@Niels\_van\_Rijn](https://discuss.elastic.co/u/Niels_van_Rijn)\ **Post date:** [June 4, 2019, 9:40am UTC](https://discuss.elastic.co/t/structure-elasticsearch-for-50-million-fields/184134/1 "2019-06-04T09:40:53Z")
Hi! I have a dataset of 43536000 fields (I can't make it smaller, I already removed fields). With normal Elasticsearch settings (1000 shards per node and 1000 fields per index) I would have to make 44 nodes in my Elasticsearch cluster. I can raise the amount of fields per index or the amount of shards per node but I read everywhere that this is not recommended. Am I missing something? I see lots success stories with large datasets. How would one structure Elasticsearch to accommodate such a large dataset? ---
**Author:** ![wangqinghuan](https://avatars.discourse-cdn.com/v4/letter/w/d26b3c/32.png) [@wangqinghuan](https://discuss.elastic.co/u/wangqinghuan)\ **Post date:** [June 4, 2019, 10:25am UTC](https://discuss.elastic.co/t/structure-elasticsearch-for-50-million-fields/184134/2 "2019-06-04T10:25:52Z")
Because fields are included in mappings and mappings are included in cluster state which will be distributed among nodes. Too many fields will lead to big traffic between nodes. Why do you need so many fields? Maybe you can restructure documents by nested document. Here is a helpful document : [https://www.elastic.co/cn/blog/found-beginner-troubleshooting#keyvalue-woes](https://www.elastic.co/cn/blog/found-beginner-troubleshooting#keyvalue-woes) ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [June 4, 2019, 10:28am UTC](https://discuss.elastic.co/t/structure-elasticsearch-for-50-million-fields/184134/3 "2019-06-04T10:28:31Z")
What do you mean by "fields"? Do you really mean fields or documents? Do you have a sample document to share? ---
**Author:** ![Niels\_van\_Rijn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niels_van_rijn/32/45639_2.png) [@Niels\_van\_Rijn](https://discuss.elastic.co/u/Niels_van_Rijn)\ **Post date:** [June 4, 2019, 10:36am UTC](https://discuss.elastic.co/t/structure-elasticsearch-for-50-million-fields/184134/4 "2019-06-04T10:36:11Z")
So my original data is XML converted to JSON [here](https://pastebin.com/dTWqQeWK) is an example, I flattened the data to [this](https://pastebin.com/JUbY1pL3) because the document had a lot of mapping errors (within the same document). All the strings in this document need to be searchable. So far I've done this by creating a large text fields with all the strings with the `copy_to` functionality ```auto PUT test { "settings": { "number_of_shards": 3, "number_of_replicas": 1 }, "mappings": { "properties" : { "full_text": { "type": "text", "store": true }, "suggest" : { "type" : "completion" } }, "dynamic_templates": [ { "strings": { "match_mapping_type": "string", "mapping": { "type": "keyword", "copy_to": ["full_text", "suggest"] } } } ] } } ``` ---
**Author:** ![Niels\_van\_Rijn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niels_van_rijn/32/45639_2.png) [@Niels\_van\_Rijn](https://discuss.elastic.co/u/Niels_van_Rijn)\ **Post date:** [June 4, 2019, 10:47am UTC](https://discuss.elastic.co/t/structure-elasticsearch-for-50-million-fields/184134/5 "2019-06-04T10:47:50Z")
"fields" as in "fields within 1 document [here](https://pastebin.com/JUbY1pL3) is an example of 1 small document but some documents have 20.000 fields. My original data is heavily nested XML [here](https://pastebin.com/K5CJuiqh) is an example of one document/xml My first step was to convert the XML to json, but the converted json had a lot of mapping errors. It would be impossible (time wise) to go through all 945 xml's to fix the mapping errors. So I flattened the json file (this is how I got so many fields). All the strings and dates need to be searchable, I tested with the `index: false` `enabled: false` and `store: false` settings but to use the highlight and prefix suggestion functionalities I have to use the default settings, if I'm not mistaken. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 4, 2019, 10:48am UTC](https://discuss.elastic.co/t/structure-elasticsearch-for-50-million-fields/184134/6 "2019-06-04T10:48:12Z")
What is causing that number of fields? Are you using some type of entity identifier as field name? ---
**Author:** ![Niels\_van\_Rijn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niels_van_rijn/32/45639_2.png) [@Niels\_van\_Rijn](https://discuss.elastic.co/u/Niels_van_Rijn)\ **Post date:** [June 4, 2019, 10:56am UTC](https://discuss.elastic.co/t/structure-elasticsearch-for-50-million-fields/184134/7 "2019-06-04T10:56:08Z")
Yes, flattening the data was the only workaround I found to tackle all the mapping errors I had. This flattening is what creates all the extra fields. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 4, 2019, 11:25am UTC](https://discuss.elastic.co/t/structure-elasticsearch-for-50-million-fields/184134/8 "2019-06-04T11:25:26Z")
Can you provide an example? Would it not be possible to create a new field containing the identifier so you can filter on this and thereby dramatically reduce the number of fields? Having entity ids in field names is generally a bad practice. ---
**Author:** ![Niels\_van\_Rijn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niels_van_rijn/32/45639_2.png) [@Niels\_van\_Rijn](https://discuss.elastic.co/u/Niels_van_Rijn)\ **Post date:** [June 4, 2019, 11:38am UTC](https://discuss.elastic.co/t/structure-elasticsearch-for-50-million-fields/184134/9 "2019-06-04T11:38:25Z")
[Here](https://pastebin.com/dTWqQeWK) is the original JSON and [here](https://pastebin.com/JUbY1pL3) is the the same document but flattened. And maybe? What would that look like if, let's say you have this data? ```auto { "toestand": { "@bwb-id": "BWBR0018883", "@inwerkingtreding": "2006-01-14", "wetgeving": { "@dtdversie": "2.0", "intitule": { "@bwb-ng-variabel-deel": "/Intitule", "meta-data": { "brondata": { "oorspronkelijk": { "publicatie": { "@effect": "nieuwe-regeling", "uitgiftedatum": { "@isodatum": "2006-01-12", "#text": "12-01-2006" } } } } } } } } } ``` ---
**Author:** ![Niels\_van\_Rijn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niels_van_rijn/32/45639_2.png) [@Niels\_van\_Rijn](https://discuss.elastic.co/u/Niels_van_Rijn)\ **Post date:** [June 7, 2019, 8:20am UTC](https://discuss.elastic.co/t/structure-elasticsearch-for-50-million-fields/184134/10 "2019-06-07T08:20:16Z")
Can you explain what you mean, I do not fully understand what you mean by "create a new field containing the identifier". What identifier do you mean? ---
**Author:** ![Niels\_van\_Rijn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niels_van_rijn/32/45639_2.png) [@Niels\_van\_Rijn](https://discuss.elastic.co/u/Niels_van_Rijn)\ **Post date:** [June 7, 2019, 8:22am UTC](https://discuss.elastic.co/t/structure-elasticsearch-for-50-million-fields/184134/11 "2019-06-07T08:22:00Z")
Any suggestions? @dadoonet How would you index a dataset like [this](https://pastebin.com/K5CJuiqh)? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 7, 2019, 10:20am UTC](https://discuss.elastic.co/t/structure-elasticsearch-for-50-million-fields/184134/12 "2019-06-07T10:20:09Z")
I do not understand from the example what is driving the creation of 50 million fields. The example you provided looks quite simple. Can you please elaborate? ---
**Author:** ![Niels\_van\_Rijn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niels_van_rijn/32/45639_2.png) [@Niels\_van\_Rijn](https://discuss.elastic.co/u/Niels_van_Rijn)\ **Post date:** [June 7, 2019, 10:41am UTC](https://discuss.elastic.co/t/structure-elasticsearch-for-50-million-fields/184134/13 "2019-06-07T10:41:49Z")
Ah of course, So I am indexing laws, **one** law has around 3800 fields (just like [this](https://pastebin.com/JUbY1pL3) example, but longer). Each law has around 50 versions that means 50 x 3800(fields) and I have 27 laws. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2019, 10:44am UTC](https://discuss.elastic.co/t/structure-elasticsearch-for-50-million-fields/184134/14 "2019-07-05T10:44:13Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.