Fresh install kibana Rejecting mapping update to [.kibana] as the final mapping would have more than 1 type: [log, doc]

zzchen · January 29, 2018, 8:08am

Hi,

[Background]
I upgrade ES from 5.6 to 6.1.2 and the kibana version is 6.1.2 (All are docker with OSS distributed version).
For some reason, i would like to use a fresh new kibana installation, so i delete '.kibana' index, then relaunch kibana to let it create its own index from scratch.

[Problem]

I cannot "Create index pattern" on kibana. After click "create index pattern" button it return 400 error at "api/saved_objects/index-pattern".
After "problem 2", i try to create the ".kibana" index by my self. And I got error:

PUT /.kibana
    {
      "error": {
        "root_cause": [
          {
            "type": "remote_transport_exception",
            "reason": "[es-master-1][10.0.0.39:9300][indices:admin/create]"
          }
        ],
        "type": "illegal_argument_exception",
        "reason": "Rejecting mapping update to [.kibana] as the final mapping would have more than 1 type: [log, doc]"
      },
      "status": 400
    }

I found there is a template "kibana_index_template:.kibana" as below:

 "kibana_index_template:.kibana": {
    "order": 0,
    "index_patterns": [
      ".kibana"
    ],
    "settings": {
      "index": {
        "number_of_shards": "1"
      }
    },
    "mappings": {
      "doc": {
        "dynamic": "strict",
        "properties": {
          "type": {
            "type": "keyword"
          },
          "updated_at": {
            "type": "date"
          },
          "config": {
            "dynamic": true,
            "properties": {
              "buildNum": {
                "type": "keyword"
              }

... IGNORED ...

If I try to delete the template "kibana_index_template:.kibana", I will success to create ".kibana" index via PUT /.kibana.

[Question]
I know there is breaking changes regard to ES 6.0 and the Removal of Mapping Types.
It seems the template created by kibana automatically is not change to the new mapping yet?

Thank you!

bhavyarm · January 29, 2018, 10:32am

Hi,

You are correct: just deleting .kibana doesn't solve this problem.
You need to use the upgrade assistant in 5.6 - reindex .kibana and then upgrade to 6.x.

Thanks,
Bhavya

zzchen · January 29, 2018, 2:12pm

Hi,

Thanks for your reply.

I might not understand the problem.

Can't I assume only install the new kibana installation on 6.1? I can discard all previous data before 5.6, just want a new one.

If I have already upgrade ES and kibana to 6.1 with deleted .kibana and "kibana_index_template:.kibana" template'. How to upgrade by upgrade assistant?

BTW. I notice Upgrade Assistant only existed in X-Pack, but I'm using OSS docker version.

Thank you!

bhavyarm · January 30, 2018, 3:40am

Hi,

We removed types from elasticsearch in 6.0. https://www.elastic.co/blog/kibana-6-removal-of-mapping-types. So if you are planning to bring over data from 5.x to 6.x - you need to upgrade to 5.6 before - get the x-pack basic (which is free) - which will add upgrade assistant to your management view:

Use it to upgrade and then migrate to 6.x. You can also manually migrate index patterns without using the upgrade assistant: https://www.elastic.co/guide/en/elasticsearch/reference/current/reindex-upgrade.html

Hope this helps.

Thanks,
Bhavya

zzchen · January 30, 2018, 4:23am

Hi,

What if I have upgrade my cluster from 5.6 to 6.1 (it seems hard to downgrade to 5.6 right now).
Does the "reindex" by manual still work for me? (perform reindex on ES 6.1).

bhavyarm · January 30, 2018, 4:29am

Not really sure. Am checking in with our ops team and will keep you posted.

Cheers,
Bhavya

zzchen · January 30, 2018, 5:38am

Thank you for your kindly help!

bhavyarm · January 30, 2018, 5:46am

Yep it is possible to migrate in 6.x manually.
https://www.elastic.co/guide/en/kibana/6.x/migrating-6.0-index.html

Good luck,
Cheers
Bhavya

zzchen · January 30, 2018, 6:07am

Hi,
Please allow me to confirm some things with you.

The document you mention is describing how to migrate Kibana index to 6.0.
The situation i encounter is i have DELETE OLD KIBANA index, and wanna a new installation kibana 6.0.

Right now, i have no kibana index to reindex anymore. It seem it's not the solution what we are finding.

Thank you!

bhavyarm · January 30, 2018, 6:43am

Hi,

.kibana in that example could be any index. Even if you have deleted .kibana index and if you bring over rest of your other indices without reindexing - the auto-recreated .kibana will fail the mapping.

If you have no data in 5.6.7(5.x) and all you want to do is to start from 6.x - then you can just start with a fresh install of elasticsearch and Kibana in 6.0. But if you have data in 5.6 - reindexing them is necessary for your ES/Kibana to come up.

Does that help? Or should I ask somebody else to look over this?

Thanks,
Bhavya

warkolm · January 30, 2018, 6:54am

What does the mapping of the .kibana index look like now?

zzchen · January 30, 2018, 7:00am

Hi,

The .kibana index cannot create due to the error: Rejecting mapping update to [.kibana] as the final mapping would have more than 1 type: [log, doc] in my first post.

The template kibana_index_template:.kibana of .kibana as below:

{
  "kibana_index_template:.kibana": {
    "order": 0,
    "index_patterns": [
      ".kibana"
    ],
    "settings": {
      "index": {
        "number_of_shards": "1"
      }
    },
    "mappings": {
      "doc": {
        "dynamic": "strict",
        "properties": {
          "type": {
            "type": "keyword"
          },
          "updated_at": {
            "type": "date"
          },
          "config": {
            "dynamic": true,
            "properties": {
              "buildNum": {
                "type": "keyword"
              }
            }
          },
          "index-pattern": {
            "properties": {
              "fieldFormatMap": {
                "type": "text"
              },
              "fields": {
                "type": "text"
              },
              "intervalName": {
                "type": "keyword"
              },
              "notExpandable": {
                "type": "boolean"
              },
              "sourceFilters": {
                "type": "text"
              },
              "timeFieldName": {
                "type": "keyword"
              },
              "title": {
                "type": "text"
              }
            }
          },
          "visualization": {
            "properties": {
              "description": {
                "type": "text"
              },
              "kibanaSavedObjectMeta": {
                "properties": {
                  "searchSourceJSON": {
                    "type": "text"
                  }
                }
              },
              "savedSearchId": {
                "type": "keyword"
              },
              "title": {
                "type": "text"
              },
              "uiStateJSON": {
                "type": "text"
              },
              "version": {
                "type": "integer"
              },
              "visState": {
                "type": "text"
              }
            }
          },
          "search": {
            "properties": {
              "columns": {
                "type": "keyword"
              },
              "description": {
                "type": "text"
              },
              "hits": {
                "type": "integer"
              },
              "kibanaSavedObjectMeta": {
                "properties": {
                  "searchSourceJSON": {
                    "type": "text"
                  }
                }
              },
              "sort": {
                "type": "keyword"
              },
              "title": {
                "type": "text"
              },
              "version": {
                "type": "integer"
              }
            }
          },
          "dashboard": {
            "properties": {
              "description": {
                "type": "text"
              },
              "hits": {
                "type": "integer"
              },
              "kibanaSavedObjectMeta": {
                "properties": {
                  "searchSourceJSON": {
                    "type": "text"
                  }
                }
              },
              "optionsJSON": {
                "type": "text"
              },
              "panelsJSON": {
                "type": "text"
              },
              "refreshInterval": {
                "properties": {
                  "display": {
                    "type": "keyword"
                  },
                  "pause": {
                    "type": "boolean"
                  },
                  "section": {
                    "type": "integer"
                  },
                  "value": {
                    "type": "integer"
                  }
                }
              },
              "timeFrom": {
                "type": "keyword"
              },
              "timeRestore": {
                "type": "boolean"
              },
              "timeTo": {
                "type": "keyword"
              },
              "title": {
                "type": "text"
              },
              "uiStateJSON": {
                "type": "text"
              },
              "version": {
                "type": "integer"
              }
            }
          },
          "url": {
            "properties": {
              "accessCount": {
                "type": "long"
              },
              "accessDate": {
                "type": "date"
              },
              "createDate": {
                "type": "date"
              },
              "url": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 2048
                  }
                }
              }
            }
          },
          "server": {
            "properties": {
              "uuid": {
                "type": "keyword"
              }
            }
          },
          "timelion-sheet": {
            "properties": {
              "description": {
                "type": "text"
              },
              "hits": {
                "type": "integer"
              },
              "kibanaSavedObjectMeta": {
                "properties": {
                  "searchSourceJSON": {
                    "type": "text"
                  }
                }
              },
              "timelion_chart_height": {
                "type": "integer"
              },
              "timelion_columns": {
                "type": "integer"
              },
              "timelion_interval": {
                "type": "keyword"
              },
              "timelion_other_interval": {
                "type": "keyword"
              },
              "timelion_rows": {
                "type": "integer"
              },
              "timelion_sheet": {
                "type": "text"
              },
              "title": {
                "type": "text"
              },
              "version": {
                "type": "integer"
              }
            }
          }
        }
      }
    },
    "aliases": {

    }
  }
}

warkolm · January 30, 2018, 7:01am

What's the actual mapping of the index that is in Elasticsearch now look like? It should give hints as to what's happening.

zzchen · January 30, 2018, 7:07am

I've deleted the .kibana index, so there is no index mapping left.

GET /.kibana
{
  "error": {
    "root_cause": [
      {
        "type": "index_not_found_exception",
        "reason": "no such index",
        "index_uuid": "_na_",
        "resource.type": "index_or_alias",
        "resource.id": ".kibana",
        "index": ".kibana"
      }
    ],
    "type": "index_not_found_exception",
    "reason": "no such index",
    "index_uuid": "_na_",
    "resource.type": "index_or_alias",
    "resource.id": ".kibana",
    "index": ".kibana"
  },
  "status": 404
}

warkolm · January 30, 2018, 7:11am

Check _cat/indices, If there is no .kibana then restart Kibana and it should create the index and the correct mapping.

If you run into the same error then check the mapping and then post it here so we can take a closer look.

zzchen · January 30, 2018, 7:24am

Hi,

I checked _cat/indices and there is no .kibana index in ES.
I restart kibana and try to create my first index pattern.

There is a HTTP 400 error in browser,

warkolm · January 30, 2018, 7:37am

Does _cat/indices show a .kibana? What does that mapping look like?
What do the Kibana and Elasticsearch logs show?

zzchen · January 30, 2018, 7:57am

curl -XGET http://localhost:9200/_cat/indices | grep '\.kibana' not returned any result, therefore the .kibana index has not been created.

ES error

es-master-0 es-master [2018-01-30T07:52:20,577][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]

Kibana error

{"type":"response","@timestamp":"2018-01-30T07:54:00Z","tags":[],"pid":1,"method":"post","statusCode":400,"req":{"url":"/api/saved_objects/index-pattern?overwrite=false","method":"post","headers":{"host":"localhost:8001","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36","content-length":"74","accept":"application/json, text/plain, */*","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,zh-TW;q=0.8,zh;q=0.7","cache-control":"no-cache","content-type":"application/json;charset=UTF-8","kbn-version":"6.1.2","origin":"http://localhost:8001","pragma":"no-cache","referer":"http://localhost:8001/api/v1/proxy/namespaces/default/services/kibana:web/app/kibana","x-forwarded-for":"127.0.0.1, 125.227.160.187","x-forwarded-uri":"/api/v1/proxy/namespaces/default/services/kibana:web/api/saved_objects/index-pattern"},"remoteAddress":"10.128.0.6","userAgent":"10.128.0.6","referer":"http://localhost:8001/api/v1/proxy/namespaces/default/services/kibana:web/app/kibana"},"res":{"statusCode":400,"responseTime":697,"contentLength":9},"message":"POST /api/saved_objects/index-pattern?overwrite=false 400 697ms - 9.0B"}
{"type":"ops","@timestamp":"2018-01-30T07:54:00Z","tags":[],"pid":1,"os":{"load":[0.77685546875,0.96240234375,0.99560546875],"mem":{"total":27402412032,"free":6320754688},"uptime":534753},"proc":{"uptime":2352.283,"mem":{"rss":95854592,"heapTotal":66719744,"heapUsed":57941368,"external":437709},"delay":0.3062770366668701},"load":{"requests":{"5601":{"total":1,"disconnects":0,"statusCodes":{"400":1}}},"concurrents":{"5601":1},"responseTimes":{"5601":{"avg":697,"max":697}},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 55.3MB uptime: 0:39:12 load: [0.78 0.96 1.00] delay: 0.306"}

It seems the template created by kibana is not apply change to the new mapping yet like the guess at my first post?

Thank you!

warkolm · January 30, 2018, 9:21am

I'm a little lost, cause if there is no .kibana and you start Kibana, then it'll create that index automatically. It should mention that in the logs.

That is a warning, not an error.

What is this?

zzchen · January 30, 2018, 1:44pm

Yes, I'm confused too.
I understand Kibana should create the .kibana index when it starts.
This is the weirdest part, no .kibana index had been created and no error message at logging.

I past more detail log for your information.

kibana startup log:
Kibana startup log (pastbin)

to check if existed .kibana

# Check 1
curl http://elasticsearch-internal:9200/.kibana
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":".kibana","index_uuid":"_na_","index":".kibana"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":".kibana","index_uuid":"_na_","index":".kibana"},"status":404}

# Check 2
curl http://elasticsearch-internal:9200/_cat/indices | grep '\.kibana'
(no result)

BTW, i use reverse proxy and configured server path, so you will see http://localhost:8001/api/v1/proxy/namespaces/default/services/kibana:web/app/kibana.

My docker run with options and environment variables:

--verbose
SERVER_BASEPATH="/api/v1/proxy/namespaces/default/services/kibana:web/app/kibana"
KIBANA_INDEX=".kibana"

Docker Image: docker.elastic.co/kibana/kibana-oss:6.1.2

Thanks!