Limit number of fields without actually having to delete anything or hampering visualisations

[elk11]

Hi,

Number of fields in my index is shooting above the limit (1000). I don't want to increase the limit any further (to avoid mapping explosion) . But rather planning on to cut down on the number of fields. There are fields something like (Not exactly same):

last_name.jacob
last_name.susan
last_name.petrice
last_name.george

It seems like nested fields may be an option, but does visualisations work if i do something like this?

last_name: { jacob:
susan:
patrice:
george:
}

If not, what's the way to reduce the number of fields without essentially deleting any fields and should still be able to query and visualise everything?

Your help is very much appreciated! Thanks.

[elk11]

Can somebody please help with this?

[Christian_Dahlqvist]

Why would you choose to structure your data that way? Why not just use "last_name": "Smith","first_name":"Susan" or something similar? Having fields created dynamically like that will generally always cause mapping explosion and be inefficient.

[elk11]

Thanks for your response

There's a requirement for me to create fields dynamically.

Having fields created dynamically like that will generally always cause mapping explosion and be inefficient.

To avoid the this, I'm planning to use nested field or something similar.

Why not just use "last_name": "Smith","first_name":"Susan" or something similar?

I need multiple entries of this kind in a single document. If you mean I could make a list like this,

"first_name": {"susan", "jacob", "petrice", "george"}
"last_name": {"smith", "rrrr", "tttt", "mmmm"}

will i be able to map the last names with the corresponding first names? If i can, does visualisation work on this kind of fields ?

[Christian_Dahlqvist]

You can created a nested mapping with the following structure:

"users": [
  {
    "first_name": "Susan", "last_name": "Smith"
  },
  {
    "first_name": "Adam", "last_name": "Jones"
  }
]

This will allow you to search on combinations of first and last name, although Kibana largely lacks support for handling nested documents. On the other hand having very large mappings is not handled well in Kibana either....

[elk11]

so I'll have choose between "mapping explosion" and "kibana support".

Given that i presently have 1300 fields in my index, which is safer to go for? Increasing field limit or using nested fields?

will this problem be solved in any kibana upgrade( current version: 5.6.2)?

[Christian_Dahlqvist]

No, I do not think upgrading will solve this issue for you. One option might be to denormalise and still one user per document, but as I do not know your use-case I can't tell whether that is an option or not. You can follow the discussion about support for nested documents in this GitHub issue (although there may also be others)

[elk11]

Thanks for your help

Going through that discussion it seems like it's not completely implemented yet. Till then i should resolve to increasing the field limit i guess.

[Christian_Dahlqvist]

As mappings are loaded and processed in the browser, having very large mappings can slow things down considerably, so if possible I would recommend rethinking the data model.

[elk11]

Thanks for the advice. Will try and cut down on some unnecessary fields.

[elk11]

I have a followup question, till i rethink my data-model and discuss it with my team, I want to increase the field limit dynamically.

Is there a way to dynamically change the fields limit? like, if number of fields increases to 1400, the limit should adapt to that without giving any error.

[Christian_Dahlqvist]

No, you need to set it to a specific value.

[elk11]

Okay, Thanks