Rejecting mapping update to [.kibana] ... [indices:admin/create]


(Daniel Wilches) #1

We are installing Kibana in our org and we are seeing the following error message:

That appears when clicking on the "No" button you can see in the screen (but it's appearing in other more important parts of the UI).

I checked DevTools in Chrome and I saw the request is failing with this message:

{
    "message":"Rejecting mapping update to [.kibana] as the final mapping would have more than 1 type: [doc, message]: [remote_transport_exception] [WARM_MASTER_MACHINES_NAME][MACHINES_IP:9300][indices:admin/create]",
    "statusCode":400,
    "error":"Bad Request"
}

I saw many people have asked about this and there are answers mentioning:

  • Do you have a .kibana index? No, we don't, I checked with _cat.
  • Do you have xpack enabled? I shouldn't have it as we don't use username+password authentication with ES. But out of excess of caution I threw xpack.security.enabled: false in kibana.yml with no luck. I also see some logs related to xpack , should I be concerned about them?

How can I troubleshoot this issue?

Thanks.


(Daniel Wilches) #2

@bhavyarm In this post you mention one needs to reindex the data for this problem to get solved, though in my case we had ES 2, and we reindexed everything into ES 6.2.4. Sometime later we upgraded to 6.4.1 without reindexing all data as that didn't seem to be necessary. May that be an issue?


(Joe Fleming) #3

If you were already running on 6.2.4, then moving to 6.4.1 shouldn't have been an issue. The error you're getting should exist in all 6.x versions, since Elasticsearch 6.x only allows a single type for any index.

Everything Kibana writes uses doc as the type value, I'm not sure why you're getting a message type as well now. Kibana did, and I think still does, have some auto-migration stuff, and that's probably where the error is coming from.

Did you in the past, or do you right now, have any third party Kibana plugins you're using? Maybe that's where the documents with the message type are coming from?


(Daniel Wilches) #4

@Joe_Fleming We don't have any 3rd party plugin for Kibana. This is actually a fresh install we are doing for testing what Kibana does. We are installing it on a different server and making it point to our ES 6.4.1 cluster.

Is there anything I should check for in Kibana's logs? Nothing catches my eye in there.
Or is there a way to debug that auto-migration utility for Kibana?

I read somewhere that Kibana should create a .kibana index, but Kibana is not creating that one either.


(Daniel Wilches) #5

@Joe_Fleming We found some information in the logs about plugins that are being loaded, though not sure if any of them is 3rd party, as we have not attempted to install anything 3rd party on this machine. We are, though, using a Chef recipe to install Kibana, and now I'm wondering if this recipe installs by default something that can be causing issues.

These are the logs we found:

[2019-02-14T22:11:51,997][INFO ][o.e.p.PluginsService     ] [MASKED_ES_CLIENT_NAME] loaded module [aggs-matrix-stats]
[2019-02-14T22:11:51,997][INFO ][o.e.p.PluginsService     ] [MASKED_ES_CLIENT_NAME] loaded module [analysis-common]
[2019-02-14T22:11:51,997][INFO ][o.e.p.PluginsService     ] [MASKED_ES_CLIENT_NAME] loaded module [ingest-common]
[2019-02-14T22:11:51,997][INFO ][o.e.p.PluginsService     ] [MASKED_ES_CLIENT_NAME] loaded module [lang-expression]
[2019-02-14T22:11:51,997][INFO ][o.e.p.PluginsService     ] [MASKED_ES_CLIENT_NAME] loaded module [lang-mustache]
[2019-02-14T22:11:51,998][INFO ][o.e.p.PluginsService     ] [MASKED_ES_CLIENT_NAME] loaded module [lang-painless]
[2019-02-14T22:11:51,998][INFO ][o.e.p.PluginsService     ] [MASKED_ES_CLIENT_NAME] loaded module [mapper-extras]
[2019-02-14T22:11:51,998][INFO ][o.e.p.PluginsService     ] [MASKED_ES_CLIENT_NAME] loaded module [parent-join]
[2019-02-14T22:11:51,998][INFO ][o.e.p.PluginsService     ] [MASKED_ES_CLIENT_NAME] loaded module [percolator]
[2019-02-14T22:11:51,998][INFO ][o.e.p.PluginsService     ] [MASKED_ES_CLIENT_NAME] loaded module [rank-eval]
[2019-02-14T22:11:51,998][INFO ][o.e.p.PluginsService     ] [MASKED_ES_CLIENT_NAME] loaded module [reindex]
[2019-02-14T22:11:51,998][INFO ][o.e.p.PluginsService     ] [MASKED_ES_CLIENT_NAME] loaded module [repository-url]
[2019-02-14T22:11:51,998][INFO ][o.e.p.PluginsService     ] [MASKED_ES_CLIENT_NAME] loaded module [transport-netty4]
[2019-02-14T22:11:51,998][INFO ][o.e.p.PluginsService     ] [MASKED_ES_CLIENT_NAME] loaded module [tribe]
[2019-02-14T22:11:51,998][INFO ][o.e.p.PluginsService     ] [MASKED_ES_CLIENT_NAME] loaded module [x-pack-core]
[2019-02-14T22:11:51,998][INFO ][o.e.p.PluginsService     ] [MASKED_ES_CLIENT_NAME] loaded module [x-pack-deprecation]
[2019-02-14T22:11:51,998][INFO ][o.e.p.PluginsService     ] [MASKED_ES_CLIENT_NAME] loaded module [x-pack-graph]
[2019-02-14T22:11:51,998][INFO ][o.e.p.PluginsService     ] [MASKED_ES_CLIENT_NAME] loaded module [x-pack-logstash]
[2019-02-14T22:11:51,999][INFO ][o.e.p.PluginsService     ] [MASKED_ES_CLIENT_NAME] loaded module [x-pack-ml]
[2019-02-14T22:11:51,999][INFO ][o.e.p.PluginsService     ] [MASKED_ES_CLIENT_NAME] loaded module [x-pack-monitoring]
[2019-02-14T22:11:51,999][INFO ][o.e.p.PluginsService     ] [MASKED_ES_CLIENT_NAME] loaded module [x-pack-rollup]
[2019-02-14T22:11:51,999][INFO ][o.e.p.PluginsService     ] [MASKED_ES_CLIENT_NAME] loaded module [x-pack-security]
[2019-02-14T22:11:51,999][INFO ][o.e.p.PluginsService     ] [MASKED_ES_CLIENT_NAME] loaded module [x-pack-sql]
[2019-02-14T22:11:51,999][INFO ][o.e.p.PluginsService     ] [MASKED_ES_CLIENT_NAME] loaded module [x-pack-upgrade]
[2019-02-14T22:11:51,999][INFO ][o.e.p.PluginsService     ] [MASKED_ES_CLIENT_NAME] loaded module [x-pack-watcher]
[2019-02-14T22:11:51,999][INFO ][o.e.p.PluginsService     ] [MASKED_ES_CLIENT_NAME] loaded plugin [discovery-ec2]
[2019-02-14T22:11:51,999][INFO ][o.e.p.PluginsService     ] [MASKED_ES_CLIENT_NAME] loaded plugin [repository-s3]
[2019-02-14T22:11:57,306][INFO ][o.e.x.s.a.s.FileRolesStore] [MASKED_ES_CLIENT_NAME] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2019-02-14T22:11:57,914][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/10898] [Main.cc@109] controller (64 bit): Version 6.4.1 (Build 1df3104bc26648) Copyright (c) 2018 Elasticsearch BV
[2019-02-14T22:11:58,358][DEBUG][o.e.a.ActionModule       ] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2019-02-14T22:11:58,421][INFO ][o.e.d.DiscoveryModule    ] [MASKED_ES_CLIENT_NAME] using discovery type [zen]

(Joe Fleming) #6

We found some information in the logs about plugins that are being loaded, though not sure if any of them is 3rd party

All apps and even large parts of Kibana are actually plugins, so that's normal. If you didn't manually install plugins, then you don't have any 3rd party ones.

That said, the discovery-ec2 and repository-s3 are unfamiliar to me. I guess the easiest way to check if you have plugins installed is just to look in the plugins path in the root of Kibana. If the directory is empty, then you don't have any 3rd party plugins.

I read somewhere that Kibana should create a .kibana index, but Kibana is not creating that one either.

It will create it, but I believe it waits until you save something in Kibana. If you create an index pattern, for example, you'll have a .kibana index. I believe that regardless of the index existing, the index template is created when you start Kibana, and that would define what types exist in the index.

Something is trying to write a document to the .kibana index with "_type": "message", that's what is causing the error you see. But if it's not some other plugin, I'm not sure what would be doing that.

It's possible, but I think less likely, that the opposite is happening... that something else is creating the .kibana index and adding a document with "_type": "message", so when Kibana tries to write to the index, doc becomes the second type. But, if you don't have any plugins installed, I can't think why that would happen.