Kibana: mapping set to strict, dynamic introduction of [references] within [doc] is not allowed

Elastic Stack Kibana
1

Hi

elastic, kibana 7.0.0 (rpm installation)
Centos7

Updated both from 6.7.1 to 7.0.0.
Cluster is green, but when browsing kibana on browser, I get:

{"message":"mapping set to strict, dynamic introduction of [references] within [doc] is not allowed: [strict_dynamic_mapping_exception] mapping set to strict, dynamic introduction of [references] within [doc] is not allowed","statusCode":400,"error":"Bad Request"}

Elasticsearch log has this:

[2019-04-11T15:32:31,084][DEBUG][o.e.a.b.TransportShardBulkAction] [dev-dc1-rk] [.kibana_1][0] failed to execute bulk item (create) index {[.kibana][_doc][config:7.0.0], source[{"config":{"buildNum":23117,"defaultIndex":"11ed3b50-4180-11e9-8c13-3da5f557b485"},"type":"config","references":[],"updated_at":"2019-04-11T12:32:31.082Z"}]}
org.elasticsearch.index.mapper.StrictDynamicMappingException: mapping set to strict, dynamic introduction of [references] within [doc] is not allowed
        at org.elasticsearch.index.mapper.DocumentParser.parseArray(DocumentParser.java:536) ~[elasticsearch-7.0.0.jar:7.0.0]
        at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:394) ~[elasticsearch-7.0.0.jar:7.0.0]
        at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:381) ~[elasticsearch-7.0.0.jar:7.0.0]
        at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:98) ~[elasticsearch-7.0.0.jar:7.0.0]
        at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:71) ~[elasticsearch-7.0.0.jar:7.0.0]
        at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:267) ~[elasticsearch-7.0.0.jar:7.0.0]
        at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:770) ~[elasticsearch-7.0.0.jar:7.0.0]
        at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:747) ~[elasticsearch-7.0.0.jar:7.0.0]
        at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:719) ~[elasticsearch-7.0.0.jar:7.0.0]

Regards
Raul

2 Likes
2

Same issue here. Came from an elastic stack 6.7.1 and just updated to 7.0.0.
Our stack runs in docker. Kibana has no additional config.

3

Encountering the same behavior, after upgrading from 6.7.0 to 7.0.0 (tar.gz).
I copied the configuration (kibana.yml) from my previous install.

I tried to find where was this reference to mapping {dynamic: "strict"} and it's located in the indexes .kibana_*

{
  ".kibana_1": {
    "aliases": {},
    "mappings": {
      "dynamic": "strict",
      "_meta": {
[ ... ]

I then stopped Kibana and removed those indexes
(WARNING : it will delete all your visualizations and dashboards. Be sure to have a backup) :

curl -XDELETE localhost:9201/.kibana_*

Just to check it's OK :

curl -XGET localhost:9201/.kibana_*
{}

But when I start Kibana, it recreates .kibana_X indexes with the attribute set to the incompatible value.

4

Same issue on Ubuntu 18.04.2
Came from 6.7.1 to 7.0.0 (apt)

5

Having the exact same issue coming from version 6.7.1.

6

Removed the .kibana_* from above uninstalled kibana, then reinstalled.
Kibana is up and running.

7

Be careful with this because this will remove all of your saved objects (searches, visualizations, dashboards, index patterns, etc..). Well at least it did for me.

Please let me know if there is a way to retrieve the saved objects because I can't find those anywhere after deleting kibana* :confused:

1 Like
8

Sorry forgot to mention that. This was a test cluster so I didn't care.

9

Same issue on RHEL
Came from 6.7.1 to 7.0.0 (yum)

10

I wonder, why anyone doesn't react from kibana's side!

This isn't some minor issue, after all this forbids us to use kibana at all.

Delete'ing .kibana_* indices is not an option, since all the settings, dashboards etc are gone after that.

Raul

1 Like
11

Yes you're right : it was only for debugging and as I have a backup, I forgot to mention that too.
I'll edit my post with a warning.

12

Strange, I have the same problem when I setup a fresh installation of elasticsearch+kibana 7.0.0 and restore my backup from my old <7.0.0 installation.
When I however restore my backup into a fresh 6.7.1 installation and then simply update to 7.0.0, it still works fine even after the update.

13

I was fortunate to have exported all the objects from a month ago. This too is my test environment.

Would you back up with a snapshot or exported objects for this specific example?

14

Glad to hear you found a backup.

As a I'm migrating from a 6.7.0 env, I copied the 6.7.0 ES data directory into the 7.0.0 ES directory.
That way, I can play with my data without any risk : if I mess it up, I just have to perform another copy.

15

Interesting : the only migration path I'm aware of is the first one you described, by copying the data directory into the freshly installed 7.0.0 ES, as I understood it from the rolling update page : https://www.elastic.co/guide/en/elasticsearch/reference/7.0/rolling-upgrades.html

Then, what do you mean by "simply update to 7.0.0" ?

16

I see, I was not precise in my wording: I'm running elasticsearch/kibana in a k8s cluster (elasticsearch running as a statefulset). When I say "simply update", I mean that I just replace the pointer to the docker images in that statefulset, the effect is that the running containers are replaced and the data directories are preserved (that's the rolling upgrade as referenced by you).
As said, this one works fo me.
What doesn't work: I have regular backups (elasticsearch snapshot API) stashed into a gcs bucket. When I restore a snapshot that was created from 6.7 directly into an empty 7.0 elasticsearch/kibana installation, then it fails with the error discussed in this thread.

17

Same issue here - elastic cloud managed environment - 6.7.1 to 7.0.0 upgrade via UI. Have just saved .kibana_*, deleted and restarted and can at least get in to the UI again now.

18

Ok, I get it now, thanks for explaining !

19

I opened bug report here, perhaps they'll review it there:

Raul

2 Likes
21

Hi,
it seems, that the old kibana_* like e.q. kibana_7 has type:doc while the new kibana_8 has type:_doc. When now using an index pattern of kibana_* you run into the strict rule.
On my testsystem I had .kibana_8 and .kibana_7 where .kibana_8 seems to be a migrated version of kibana_7. I bypassed this issue by executing

curl -XGET localhost:9200/.kibana_7/_search >kibana_7
curl -XDELETE localhost:9200/.kibana_7

After Kibana restart I have access, but lost the Kibana Index patterns and the dashboards. I will wait for a fix before upgrading my prod system.