How to map fields with dynamic keys

Elastic Stack Elasticsearch
1

Hi guys, I need to add some dynamic fields into my documents but those fields have dynamic keys like this example:

[
{
"loc_42": 0.5,
"loc_21": 0.7
},
{
"loc_42": 0.5,
"loc_15": 0.8,
"loc_18": 0.2,
"loc_2": 0.1
},
{
"loc_421": 0.5,
"loc_11": 0.8,
"loc_13": 0.2,
"loc_20": 0.1
},
....
]

The nature of my keys is dynamic, it may include any id number. The problem of just adding these documents is to inflate my mapping schema with a very large number of different keys. I tried to use dynamic templates but I had the same problem.

Is possible to add those fields to my documents, keep them searchable but without having them on mappings?

Thank you!

1 Like
2

You can use trailing wildcards in templates, so "loc_*": "float" or similar.

1 Like
3

Hey @warkolm, thank you for the reply. I tried to do it before, but with dynamic templates , I still have all the keys in mapping schema. For the dynamic template you suggested, after indexing my documents, I got it:

  GET my_index/_mappings

  {
  "my_index": {
    "mappings": {
      "my_type": {
        "dynamic_templates": [
          {
            "dynamic_keys": {
              "mapping": {
                "type": "float"
              },
              "match": "loc_*"
            }
          }
        ],
        "properties": {
          "loc_11": {
            "type": "float"
          },
          "loc_13": {
            "type": "float"
          },
          "loc_15": {
            "type": "float"
          },
          "loc_18": {
            "type": "float"
          },
          "loc_2": {
            "type": "float"
          },
          "loc_20": {
            "type": "float"
          },

I would like to avoid that because the number of dynamic keys I may generate in my real scenario is very large.

4

What are you after then?

5

Indexing all this data on Elasticsearch but without flooding my mapping schema with a lot of keys/fields. In Solr, for instance, I could create a dynamic field loc_*: float without having each possible key value in my schema. I would like to have something like this.

I have total control over the indexing, so I can represent keys differently if necessary.

6

So you want to merge them all into one field name?

7

It could be. What I need is to have these 'dynamic fields' in my documents, to boost by their values in queries or to filter by these keys. However, I want to have them mapped to only one field (or one field pattern) in mappings.

8

Ahh ok.

Well you can set one definition in the template, but ES will still keep all the individual fields in the mappings irrespectively. You cannot change this.

9

Ok, thank you very much for your help.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.