Kibana 7.9 Index Patterns Do Not Pick Up Source Fields

[SKRYMr (Alessio Boiardi)]

Hello, I was trying to set up some visualizations for our data so I created an index pattern to match all our production indices.

What happens is that the indices are matched correctly but only "root" fields are picked up and made available for Visualizations and Dashboards, namely _index, _id, _score, _type and _source.
All the fields we defined with the PUT Mapping API inside of the _source field are invisible.

As mentioned in the title we are using Kibana 7.9 and our indices' mappings are defined at the time of creation, although they are kept dynamic to allow for further modification down the line.
We are perfectly able to search, aggregate and use the fields inside of the _source with APIs and Dev Tools, but they are inaccessible by Visualizations since they don't appear in the Index Pattern.

I tried various combinations of creating indices before/after Index Pattern definition and updating mappings, I also tried creating an index wihtout mappings and indexing a full document to let it pick them up by itself but that didn't work either.

I've seen a couple other questions about this but they were all closed due to inactivity.
There might be something trivial that I'm missing but it seems to me this part is very poorly documented so I had to do some trial-and-error. I hope to fix this soon because I would love to fully exploit the Visualizations, thanks.

[flash1293 (Joe Reuter)]

In 7.9 the index pattern in Kibana caches the field list from the mappings of your indices. To update, go to the index pattern management page and press the "refresh" button in the top right (as described here: Index patterns and fields | Kibana Guide [7.9] | Elastic )

In 7.11, this is not necessary anymore and the index pattern will automatically pick up new fields from the mapping.

I might misunderstand you though, so if you know about this option and you tried it already - you mention _source a few times, but index patterns only list indexed fields, not _source fields. Could you share the mapping of your index and a screenshot of the index pattern page in management?

[SKRYMr (Alessio Boiardi)]

Thanks for the reply, so if I understand what you're saying I can't access fields inside the _source field in the index pattern? Is there a way around this, maybe by aliasing fields in the source to the root of document?

[flash1293 (Joe Reuter)]

Is there a way around this, maybe by aliasing fields in the source to the root of document?

That's possible in 7.12 which was released yesterday. It's called runtime fields, you can specify them as part of the index pattern.

Be aware that they are not as performant as properly indexed fields which should be the default for known fields.

[flash1293 (Joe Reuter)]

We are perfectly able to search, aggregate and use the fields inside of the _source with APIs and Dev Tools

Fields within _source are not aggregatable nor searchable. We might talk about slightly different things here. It would help a lot if you would copy/paste your mapping and a screenshot of the index pattern in the UI.

[SKRYMr (Alessio Boiardi)]

{
    "mappings": {
      "dynamic": "true",
      "_source": {
        "enabled": "true"
      },
      "properties": {
        "_text_vector": {
          "type": "dense_vector",
          "dims": 512
        },
        "_tags": {
          "type": "keyword"
        },
        "_text": {
          "type": "text"
        },
        "_tokens": {
          "type": "keyword"
        },
        "_doc_type": {
          "type": "keyword"
        },
        "_all": {
          "properties": {
            "ext": {
              "type": "keyword"
            },
            "filename": {
              "type": "keyword"
            },
            "info": {
              "properties": {
                "native_language": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "identification": {
                  "properties": {
                    "name": {
                      "type": "keyword"
                    },
                    "surname": {
                      "type": "keyword"
                    },
                    "email": {
                      "type": "keyword"
                    },
                    "jobtitle": {
                      "type": "text",
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      }
                    },
                    "address": {
                      "properties": {
                        "city": {
                          "type": "text",
                          "fields": {
                            "keyword": {
                              "type": "keyword",
                              "ignore_above": 256
                            }
                          }
                        },
                        "country": {
                          "type": "text",
                          "fields": {
                            "keyword": {
                              "type": "keyword",
                              "ignore_above": 256
                            }
                          }
                        },
                        "country_code": {
                          "type": "keyword"
                        },
                        "county": {
                          "type": "text",
                          "fields": {
                            "keyword": {
                              "type": "keyword",
                              "ignore_above": 256
                            }
                          }
                        },
                        "coordinates": {
                          "type": "geo_point"
                        },
                        "postal_code": {
                          "type": "keyword"
                        },
                        "region": {
                          "type": "text",
                          "fields": {
                            "keyword": {
                              "type": "keyword",
                              "ignore_above": 256
                            }
                          }
                        },
                        "street_address": {
                          "type": "text",
                          "fields": {
                            "keyword": {
                              "type": "keyword",
                              "ignore_above": 256
}}}}}}}}}}}}}}

This is a snippet taken from our mappings, the full JSON is approx. 650 lines long, with a fair amount of nested fields and dense_vector fields. In this particular instance I wished to create a Map Visualization to show the geographical distribution of our users through _all.info.identification.address.coordinates.

As you can see when I create the index pattern to match our users indices, the only fields that are picked up are metafields created by Elastic itself.

[flash1293 (Joe Reuter)]

Thanks, this was very helpful. The problem is your field names are all starting with an underscore which is reserved for internal use.

See also this thread: https://github.com/elastic/kibana/issues/2551

The official answer is:

Hi,
We reserve fields beginning with _ for use within elasticsearch, outside of the _source of a document. While elasticsearch does not enforce this rule, we do in Kibana.

By changing your mapping to all instead of _all (and so on for your other field names), Kibana should work correctly.

[SKRYMr (Alessio Boiardi)]

Thanks, this is really disconcerting to find out now.

I share the sentiment of the slew of people who commented on the thread you linked, it's a weird limitation and I hope you will consider at least making it clear in the docs if not removing it altogether.

[flash1293 (Joe Reuter)]

I can definitely see the frustration. I raised this issue internally with the responsible team.