# How to design ES for user-defined custom fields **URL:** \ **Category:** Elasticsearch\ **Created:** [August 2, 2018, 12:46am UTC](https://discuss.elastic.co/t/how-to-design-es-for-user-defined-custom-fields/142663 "2018-08-02T00:46:44Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Redkin](https://avatars.discourse-cdn.com/v4/letter/r/9de0a6/32.png) [@Redkin](https://discuss.elastic.co/u/Redkin)\ **Post date:** [August 2, 2018, 12:46am UTC](https://discuss.elastic.co/t/how-to-design-es-for-user-defined-custom-fields/142663/1 "2018-08-02T00:46:44Z")
Let's say I have an index called "Contacts" Any contact belongs to a user, it'll have custom variables like this: ``` { user_id: 684498, name: "John Doe", custom_age: 27, custom_hobbies: ['painting'], custom_source: 'flyers' } ``` Users should be able to drill down by custom variables. Each user can have from a few contacts to 1M contacts. There's a potential for unlimited custom variables per contact, each user uses different custom variables. I was thinking of increasing the `index.mapping.total_fields.limit` which is 2000 by default - but that will never be enough, if you sum up the number of different custom fields between different users, it'd amount to hundreds of thousands if not millions. I was also thinking of creating an index per user, but I heard there's [performance concerns](https://github.com/elastic/elasticsearch/issues/8425). How can we do this right? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [August 2, 2018, 6:59am UTC](https://discuss.elastic.co/t/how-to-design-es-for-user-defined-custom-fields/142663/2 "2018-08-02T06:59:08Z")
If you share that as you proposed within the same index, imagine the following situation. A user A index: ```auto { user_id: 684498, name: "John Doe", custom_foo: "2018-08-01" } ``` Then the user B index: ```auto { user_id: 684498, name: "John Doe", custom_foo: "Hello world" } ``` 💥 As you can imagine. That is not going to work. You could think also of the following structure: ```auto { user_id: 684498, name: "John Doe", custom: [ { "name": "foo", "value": "Hello world" } ] } ``` But this will have some consequences: - You must use `nested` type which is coming with some drawbacks - You can't have real types for your fields as everything will be indexed as `text` If you can know what are the data types used by your user, you can think of something like: ```auto { user_id: 684498, name: "John Doe", custom_text: [ { "name": "foo", "value": "Hello world" } ], custom_date: [ { "name": "foo", "value": "2018-08-02" } ] } ``` Or like this: ```auto { user_id: 684498, name: "John Doe", custom: { dt_foo: "2018-08-01", st_foo: "Hello world" } } ``` Where `dt` and `st` are prefix for dates and strings which can be used by the dynamic templates. If you can't know what the users will enter and leave that entirely to them and that you need to index this data (otherwise you can just ask elasticsearch to ignore anything under `custom`), the I believe the best choice is to have one index per user with the drawbacks you are already aware of. Not an easy decision to make but I hope this will gives you some clues. ---
**Author:** ![Redkin](https://avatars.discourse-cdn.com/v4/letter/r/9de0a6/32.png) [@Redkin](https://discuss.elastic.co/u/Redkin)\ **Post date:** [August 2, 2018, 1:28pm UTC](https://discuss.elastic.co/t/how-to-design-es-for-user-defined-custom-fields/142663/3 "2018-08-02T13:28:13Z")
Thanks @dadoonet - this is helpful. What about an index per user method - is it practical when you have an abundance of users? (100k-1M users) If it's not, can the ElasticCloud's PaaS handle that amount of indexes even though a lot of them will hold little data? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [August 2, 2018, 3:50pm UTC](https://discuss.elastic.co/t/how-to-design-es-for-user-defined-custom-fields/142663/4 "2018-08-02T15:50:31Z")
No. I believe this strategy might require much more nodes. But may be @jpountz can share his thoughts on a Lucene level. ---
**Author:** ![ddorian43](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ddorian43/32/36093_2.png) [@ddorian43](https://discuss.elastic.co/u/ddorian43)\ **Post date:** [August 2, 2018, 4:01pm UTC](https://discuss.elastic.co/t/how-to-design-es-for-user-defined-custom-fields/142663/5 "2018-08-02T16:01:17Z")
Create `field_0_int`, `field_0_date`, `field_0_string` etc, and allow each user to have, say, 500 custom fields for each type, and create another table/index to map custom fields to the names for each user (or just a field in the current users table) and it's done. I don't think the elastic service doesn't have some magic sauce code to support billion indexes. ---
**Author:** ![Redkin](https://avatars.discourse-cdn.com/v4/letter/r/9de0a6/32.png) [@Redkin](https://discuss.elastic.co/u/Redkin)\ **Post date:** [August 6, 2018, 3:48pm UTC](https://discuss.elastic.co/t/how-to-design-es-for-user-defined-custom-fields/142663/6 "2018-08-06T15:48:41Z")
I see, thanks for the responsiveness! What is the average/rough overhead of an empty index with like 10 mapping fields? because I heard the overhead is linear in other forums. I'm just trying to understand a rough idea about the cost/scalability if we went with that idea. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 3, 2018, 3:48pm UTC](https://discuss.elastic.co/t/how-to-design-es-for-user-defined-custom-fields/142663/7 "2018-09-03T15:48:42Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.