Nested mapping for map with high cardinality keys

Elastic Stack Elasticsearch
1

I have a difficult json structure I am trying to map and index in elasticsearch. Unfortunately I am not able to restructure this json to make it more palatable for indexing.

The primary challenge is essentially a Map<UUID, Object>, .e.g

...
"simple": "value",
"number": 5513,
"someInfos": {
"D3BAE30D-2548-488F-BA60-5C12B30C1497" : {
"data" : "I want to index",
"more" : "predictable struture",
"etc" : "and so on"
},
"D29F6A0B-A765-48F0-A2D6-D5FE8CEF14D6" : {
"data" : "more stuff to index",
"youget" : "the idea here"
}
}

If I just naively try to index without an explicit mapping, the high cardinality of uuid's in the map causes Limit of total fields [1000] in index [example] has been exceeded.

I am currently getting around this problem by setting these maps "enabled": false, but I will need these values to be indexed and searchable. Ideally, I would be able to index each of these map entries as nested docs, using the UUID as the doc id for the nested doc.

While searching, I found this topic that sounded promising: How to map a dynamic map of key values? , but the nested doc link it refers to has been relocated in 7.5.0, and the current docs only make mention of nested ARRAYs of objects, not maps.

Is there a way to accomplish this via mappings or dynamic templates?

TIA.

2

Bump. Is this just something elasticsearch fundamentally cannot do due to how it stores/flattens keys?

3

Using the UUID as a field name is tricky, as you have already encountered. How about a nested array based data structure like

"someInfos" : [
  {
    "uuid" : "D3BAE30D-2548-488F-BA60-5C12B30C1497",
    ... other fields ...
  },
  {
    "uuid" : "D29F6A0B-A765-48F0-A2D6-D5FE8CEF14D6",
    ... other fields ...
  }
]

this way you would not suffer from the field level explosion. One more hint: If you want to search across more than one field within a single array element, you should go with the nested datatype

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.