# Kibana 7.9 Index Patterns Do Not Pick Up Source Fields **URL:** \ **Category:** Kibana\ **Created:** [March 24, 2021, 1:50pm UTC](https://discuss.elastic.co/t/kibana-7-9-index-patterns-do-not-pick-up-source-fields/268220 "2021-03-24T13:50:16Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![SKRYMr](https://avatars.discourse-cdn.com/v4/letter/s/c37758/32.png) [@SKRYMr](https://discuss.elastic.co/u/SKRYMr)\ **Post date:** [March 24, 2021, 1:50pm UTC](https://discuss.elastic.co/t/kibana-7-9-index-patterns-do-not-pick-up-source-fields/268220/1 "2021-03-24T13:50:16Z")
Hello, I was trying to set up some visualizations for our data so I created an index pattern to match all our production indices. What happens is that the indices are matched correctly but only "root" fields are picked up and made available for Visualizations and Dashboards, namely \_index, \_id, \_score, \_type and \_source. All the fields we defined with the PUT Mapping API inside of the \_source field are invisible. As mentioned in the title we are using Kibana 7.9 and our indices' mappings are defined at the time of creation, although they are kept dynamic to allow for further modification down the line. We are perfectly able to search, aggregate and use the fields inside of the \_source with APIs and Dev Tools, but they are inaccessible by Visualizations since they don't appear in the Index Pattern. I tried various combinations of creating indices before/after Index Pattern definition and updating mappings, I also tried creating an index wihtout mappings and indexing a full document to let it pick them up by itself but that didn't work either. I've seen a couple other questions about this but they were all closed due to inactivity. There might be something trivial that I'm missing but it seems to me this part is very poorly documented so I had to do some trial-and-error. I hope to fix this soon because I would love to fully exploit the Visualizations, thanks. ---
**Author:** ![flash1293](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/flash1293/32/41227_2.png) [@flash1293](https://discuss.elastic.co/u/flash1293)\ **Post date:** [March 24, 2021, 5:06pm UTC](https://discuss.elastic.co/t/kibana-7-9-index-patterns-do-not-pick-up-source-fields/268220/2 "2021-03-24T17:06:10Z")
In 7.9 the index pattern in Kibana caches the field list from the mappings of your indices. To update, go to the index pattern management page and press the "refresh" button in the top right (as described here: [Index patterns and fields | Kibana Guide [7.9] | Elastic](https://www.elastic.co/guide/en/kibana/7.9/managing-fields.html#_manage_your_index_pattern) ) In 7.11, this is not necessary anymore and the index pattern will automatically pick up new fields from the mapping. I might misunderstand you though, so if you know about this option and you tried it already - you mention `_source` a few times, but index patterns only list indexed fields, not `_source` fields. Could you share the mapping of your index and a screenshot of the index pattern page in management? ---
**Author:** ![SKRYMr](https://avatars.discourse-cdn.com/v4/letter/s/c37758/32.png) [@SKRYMr](https://discuss.elastic.co/u/SKRYMr)\ **Post date:** [March 24, 2021, 5:20pm UTC](https://discuss.elastic.co/t/kibana-7-9-index-patterns-do-not-pick-up-source-fields/268220/3 "2021-03-24T17:20:42Z")
Thanks for the reply, so if I understand what you're saying I can't access fields inside the \_source field in the index pattern? Is there a way around this, maybe by aliasing fields in the source to the root of document? ---
**Author:** ![flash1293](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/flash1293/32/41227_2.png) [@flash1293](https://discuss.elastic.co/u/flash1293)\ **Post date:** [March 24, 2021, 9:17pm UTC](https://discuss.elastic.co/t/kibana-7-9-index-patterns-do-not-pick-up-source-fields/268220/4 "2021-03-24T21:17:08Z")
> Is there a way around this, maybe by aliasing fields in the source to the root of document? That's possible in 7.12 which was released yesterday. It's called runtime fields, you can specify them as part of the index pattern. Be aware that they are not as performant as properly indexed fields which should be the default for known fields. ---
**Author:** ![flash1293](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/flash1293/32/41227_2.png) [@flash1293](https://discuss.elastic.co/u/flash1293)\ **Post date:** [March 24, 2021, 9:19pm UTC](https://discuss.elastic.co/t/kibana-7-9-index-patterns-do-not-pick-up-source-fields/268220/5 "2021-03-24T21:19:19Z")
> We are perfectly able to search, aggregate and use the fields inside of the \_source with APIs and Dev Tools Fields within \_source are not aggregatable nor searchable. We might talk about slightly different things here. It would help a lot if you would copy/paste your mapping and a screenshot of the index pattern in the UI. ---
**Author:** ![SKRYMr](https://avatars.discourse-cdn.com/v4/letter/s/c37758/32.png) [@SKRYMr](https://discuss.elastic.co/u/SKRYMr)\ **Post date:** [March 25, 2021, 9:09am UTC](https://discuss.elastic.co/t/kibana-7-9-index-patterns-do-not-pick-up-source-fields/268220/6 "2021-03-25T09:09:33Z")
```auto { "mappings": { "dynamic": "true", "_source": { "enabled": "true" }, "properties": { "_text_vector": { "type": "dense_vector", "dims": 512 }, "_tags": { "type": "keyword" }, "_text": { "type": "text" }, "_tokens": { "type": "keyword" }, "_doc_type": { "type": "keyword" }, "_all": { "properties": { "ext": { "type": "keyword" }, "filename": { "type": "keyword" }, "info": { "properties": { "native_language": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "identification": { "properties": { "name": { "type": "keyword" }, "surname": { "type": "keyword" }, "email": { "type": "keyword" }, "jobtitle": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "address": { "properties": { "city": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "country": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "country_code": { "type": "keyword" }, "county": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "coordinates": { "type": "geo_point" }, "postal_code": { "type": "keyword" }, "region": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "street_address": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 }}}}}}}}}}}}}} ``` This is a snippet taken from our mappings, the full JSON is approx. 650 lines long, with a fair amount of nested fields and dense\_vector fields. In this particular instance I wished to create a Map Visualization to show the geographical distribution of our users through `_all.info.identification.address.coordinates`. ![Screenshot from 2021-03-25 09-15-08](https://us1.discourse-cdn.com/elastic/original/3X/7/2/722e71db6321b77f2e8902f430e1c829ab35fb16.png) As you can see when I create the index pattern to match our users indices, the only fields that are picked up are metafields created by Elastic itself. ---
**Author:** ![flash1293](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/flash1293/32/41227_2.png) [@flash1293](https://discuss.elastic.co/u/flash1293)\ **Post date:** [March 25, 2021, 10:34am UTC](https://discuss.elastic.co/t/kibana-7-9-index-patterns-do-not-pick-up-source-fields/268220/7 "2021-03-25T10:34:20Z")
Thanks, this was very helpful. The problem is your field names are all starting with an underscore which is reserved for internal use. See also this thread: [https://github.com/elastic/kibana/issues/2551](https://github.com/elastic/kibana/issues/2551) The official answer is: > Hi, > We reserve fields beginning with \_ for use within elasticsearch, outside of the \_source of a document. While elasticsearch does not enforce this rule, we do in Kibana. By changing your mapping to `all` instead of `_all` (and so on for your other field names), Kibana should work correctly. ---
**Author:** ![SKRYMr](https://avatars.discourse-cdn.com/v4/letter/s/c37758/32.png) [@SKRYMr](https://discuss.elastic.co/u/SKRYMr)\ **Post date:** [March 25, 2021, 10:59am UTC](https://discuss.elastic.co/t/kibana-7-9-index-patterns-do-not-pick-up-source-fields/268220/8 "2021-03-25T10:59:35Z")
Thanks, this is really disconcerting to find out now. I share the sentiment of the slew of people who commented on the thread you linked, it's a weird limitation and I hope you will consider at least making it clear in the docs if not removing it altogether. ---
**Author:** ![flash1293](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/flash1293/32/41227_2.png) [@flash1293](https://discuss.elastic.co/u/flash1293)\ **Post date:** [March 25, 2021, 11:01am UTC](https://discuss.elastic.co/t/kibana-7-9-index-patterns-do-not-pick-up-source-fields/268220/9 "2021-03-25T11:01:16Z")
I can definitely see the frustration. I raised this issue internally with the responsible team. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 22, 2021, 11:01am UTC](https://discuss.elastic.co/t/kibana-7-9-index-patterns-do-not-pick-up-source-fields/268220/10 "2021-04-22T11:01:41Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.