Kibana cannot perform migrations due to validation error

dbernard · July 14, 2021, 2:08pm

Kibana is unable to perform migrations and then reports it cannot connect to elastic search.
elasticsearch is running locally.

Upgrading to 7.11.2 elasticsearch and kibana.

"type":"log","@timestamp":"2021-07-14T14:02:53+00:00","tags":["info","savedobjects-service"],"pid":830523,"message":"Starting saved objects migrations"}
{"type":"log","@timestamp":"2021-07-14T14:02:53+00:00","tags":["info","savedobjects-service"],"pid":830523,"message":"Creating index .kibana_task_manager_1."}
{"type":"log","@timestamp":"2021-07-14T14:02:53+00:00","tags":["warning","savedobjects-service"],"pid":830523,"message":"Unable to connect to Elasticsearch. Error: validation_exception"}
{"type":"log","@timestamp":"2021-07-14T14:02:53+00:00","tags":["fatal","root"],"pid":830523,"message":"ResponseError: validation_exception\n    at onBody (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:333:23)\n    at IncomingMessage.onEnd (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:260:11)\n    at IncomingMessage.emit (events.js:327:22)\n    at endReadableNT (internal/streams/readable.js:1327:12)\n    at processTicksAndRejections (internal/process/task_queues.js:80:21) {\n  meta: {\n    body: { error: [Object], status: 400 },\n    statusCode: 400,\n    headers: {\n      'content-type': 'application/json; charset=UTF-8',\n      'content-length': '379'\n    },\n    meta: {\n      context: null,\n      request: [Object],\n      name: 'elasticsearch-js',\n      connection: [Object],\n      attempts: 0,\n      aborted: false\n    }\n  }\n}"}
{"type":"log","@timestamp":"2021-07-14T14:02:53+00:00","tags":["info","plugins-system"],"pid":830523,"message":"Stopping all plugins."}
{"type":"log","@timestamp":"2021-07-14T14:02:53+00:00","tags":["info","plugins","monitoring","monitoring","kibana-monitoring"],"pid":830523,"message":"Monitoring stats collection is stopped"}
{"type":"log","@timestamp":"2021-07-14T14:02:53+00:00","tags":["info","savedobjects-service"],"pid":830523,"message":"Creating index .kibana_2."}
{"type":"log","@timestamp":"2021-07-14T14:02:53+00:00","tags":["warning","savedobjects-service"],"pid":830523,"message":"Unable to connect to Elasticsearch. Error: validation_exception"}

cheiligers · July 14, 2021, 7:12pm

@dbernard please see this discuss post that walks through some debugging steps. If that doesn't solve your issue, could you provide more verbose logs please?

dbernard · July 15, 2021, 12:09am

There are a bunch of start up errors I'll look at.
kibana 6.2.4-1 installation was inadvertently upgraded to 7.10 then 7.12 by accident.

{"type":"log","@timestamp":"2021-07-15T00:19:26+00:00","tags":["warning","plugins","licensing"],"pid":1837423,"message":"License information could not be obtained from Elasticsearch due to Error: Cluster client cannot be used after it has been closed. error"}
{"type":"log","@timestamp":"2021-07-15T00:19:56+00:00","tags":["warning","plugins","licensing"],"pid":1837423,"message":"License information could not be obtained from Elasticsearch due to Error: Cluster client cannot be used after it has been closed. error"}
{"type":"log","@timestamp":"2021-07-15T00:20:26+00:00","tags":["warning","plugins","licensing"],"pid":1837423,"message":"License information could not be obtained from Elasticsearch due to Error: Cluster client cannot be used after it has been closed. error"}
{"type":"log","@timestamp":"2021-07-15T00:20:56+00:00","tags":["warning","plugins","licensing"],"pid":1837423,"message":"License information could not be obtained from Elasticsearch due to Error: Cluster client cannot be used after it has been closed. error"}
{"type":"log","@timestamp":"2021-07-15T00:21:10+00:00","tags":["info","plugins-system"],"pid":1837423,"message":"Stopping all plugins."}
{"type":"log","@timestamp":"2021-07-15T00:21:17+00:00","tags":["info","plugins-service"],"pid":1845073,"message":"Plugin \"visTypeXy\" is disabled."}
{"type":"log","@timestamp":"2021-07-15T00:21:17+00:00","tags":["warning","config","deprecation"],"pid":1845073,"message":"Setting [elasticsearch.username] to \"elastic\" is deprecated. You should use the \"kibana_system\" user instead."}
{"type":"log","@timestamp":"2021-07-15T00:21:17+00:00","tags":["warning","config","deprecation"],"pid":1845073,"message":"Config key [monitoring.cluster_alerts.email_notifications.email_address] will be required for email notifications to work in 8.0.\""}
{"type":"log","@timestamp":"2021-07-15T00:21:17+00:00","tags":["warning","config","deprecation"],"pid":1845073,"message":"Setting [monitoring.username] to \"elastic\" is deprecated. You should use the \"kibana_system\" user instead."}
{"type":"log","@timestamp":"2021-07-15T00:21:18+00:00","tags":["info","plugins-system"],"pid":1845073,"message":"Setting up [102] plugins: [taskManager,licensing,globalSearch,globalSearchProviders,code,usageCollection,xpackLegacy,telemetryCollectionManager,telemetry,telemetryCollectionXpack,kibanaUsageCollection,securityOss,newsfeed,mapsLegacy,kibanaLegacy,translations,share,legacyExport,embeddable,uiActionsEnhanced,expressions,charts,esUiShared,bfetch,data,home,observability,console,consoleExtensions,apmOss,searchprofiler,painlessLab,grokdebugger,management,indexPatternManagement,advancedSettings,fileUpload,savedObjects,visualizations,visTypeVislib,visTypeVega,visTypeTimeseries,visTypeTimeseriesEnhanced,visTypeTimelion,features,licenseManagement,dataEnhanced,watcher,canvas,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeMarkdown,tileMap,regionMap,mapsOss,lensOss,inputControlVis,graph,timelion,dashboard,dashboardEnhanced,visualize,discover,discoverEnhanced,savedObjectsManagement,spaces,security,savedObjectsTagging,maps,lens,reporting,lists,encryptedSavedObjects,dashboardMode,cloud,upgradeAssistant,snapshotRestore,fleet,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,enterpriseSearch,ml,beatsManagement,wazuh,transform,ingestPipelines,eventLog,actions,alerts,triggersActionsUi,stackAlerts,securitySolution,case,infra,monitoring,logstash,apm,uptime]"}
{"type":"log","@timestamp":"2021-07-15T00:21:18+00:00","tags":["info","plugins","taskManager"],"pid":1845073,"message":"TaskManager is identified by the Kibana UUID: d288334a-6d1f-4f72-ae5d-145112a02759"}
{"type":"log","@timestamp":"2021-07-15T00:21:18+00:00","tags":["warning","plugins","security","config"],"pid":1845073,"message":"Session cookies will be transmitted over insecure connections. This is not recommended."}
{"type":"log","@timestamp":"2021-07-15T00:21:18+00:00","tags":["warning","plugins","reporting","config"],"pid":1845073,"message":"Chromium sandbox provides an additional layer of protection, but is not supported for Linux CentOS 7.9.2009 OS. Automatically setting 'xpack.reporting.capture.browser.chromium.disableSandbox: true'."}
{"type":"log","@timestamp":"2021-07-15T00:21:18+00:00","tags":["info","plugins","monitoring","monitoring"],"pid":1845073,"message":"config sourced from: production cluster"}
{"type":"log","@timestamp":"2021-07-15T00:21:18+00:00","tags":["info","savedobjects-service"],"pid":1845073,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
{"type":"log","@timestamp":"2021-07-15T00:21:19+00:00","tags":["info","savedobjects-service"],"pid":1845073,"message":"Starting saved objects migrations"}
{"type":"log","@timestamp":"2021-07-15T00:21:19+00:00","tags":["info","savedobjects-service"],"pid":1845073,"message":"Creating index .kibana_task_manager_1."}
{"type":"log","@timestamp":"2021-07-15T00:21:19+00:00","tags":["warning","savedobjects-service"],"pid":1845073,"message":"Unable to connect to Elasticsearch. Error: validation_exception"}
{"type":"log","@timestamp":"2021-07-15T00:21:19+00:00","tags":["fatal","root"],"pid":1845073,"message":"ResponseError: validation_exception\n    at onBody (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:333:23)\n    at IncomingMessage.onEnd (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:260:11)\n    at IncomingMessage.emit (events.js:327:22)\n    at endReadableNT (internal/streams/readable.js:1327:12)\n    at processTicksAndRejections (internal/process/task_queues.js:80:21) {\n  meta: {\n    body: { error: [Object], status: 400 },\n    statusCode: 400,\n    headers: {\n      'content-type': 'application/json; charset=UTF-8',\n      'content-length': '379'\n    },\n    meta: {\n      context: null,\n      request: [Object],\n      name: 'elasticsearch-js',\n      connection: [Object],\n      attempts: 0,\n      aborted: false\n    }\n  }\n}"}
{"type":"log","@timestamp":"2021-07-15T00:21:19+00:00","tags":["info","plugins-system"],"pid":1845073,"message":"Stopping all plugins."}
{"type":"log","@timestamp":"2021-07-15T00:21:19+00:00","tags":["info","plugins","monitoring","monitoring","kibana-monitoring"],"pid":1845073,"message":"Monitoring stats collection is stopped"}
{"type":"log","@timestamp":"2021-07-15T00:21:19+00:00","tags":["info","savedobjects-service"],"pid":1845073,"message":"Creating index .kibana_1."}
{"type":"log","@timestamp":"2021-07-15T00:21:19+00:00","tags":["warning","savedobjects-service"],"pid":1845073,"message":"Unable to connect to Elasticsearch. Error: validation_exception"}
{"type":"log","@timestamp":"2021-07-15T00:21:48+00:00","tags":["warning","plugins","licensing"],"pid":1845073,"message":"License information could not be obtained from Elasticsearch due to Error: Cluster client cannot be used after it has been closed. error"}
{"type":"log","@timestamp":"2021-07-15T00:22:18+00:00","tags":["warning","plugins","licensing"],"pid":1845073,"message":"License information could not be obtained from Elasticsearch due to Error: Cluster client cannot be used after it has been closed. error"}
{"type":"log","@timestamp":"2021-07-15T00:22:48+00:00","tags":["warning","plugins","licensing"],"pid":1845073,"message":"License information could not be obtained from Elasticsearch due to Error: Cluster client cannot be u

dbernard · July 15, 2021, 12:13am

kibana.yml is minimal

server.port: 5601
server.name: "elk101"
elasticsearch.hosts: [ "http://localhost:9200" ]
elasticsearch.requestTimeout: 3000000 

server.host: "0.0.0.0"

xpack.security.enabled: true
elasticsearch.username: elastic
elasticsearch.password: changeme

uiSettings.overrides.defaultRoute: "/app/wazuh"
elasticsearch.ssl.verificationMode: none

dbernard · July 15, 2021, 3:13pm

Looks like the problem was with the upgraded elasticsearch not having a large enough maximum shards per node.

system · August 12, 2021, 3:14pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Error: Unable to complete saved object migrations for the [.kibana_task_manager] index Elasticsearch	6	8619	August 20, 2022
Kibana error Kibana	2	266	August 2, 2022
.kibana index failed Kibana	9	913	November 18, 2022
Kibana upgrade from 7.12.0 to 7.13.1 is failing Kibana	9	1899	July 14, 2021
Kibana is not ready in 7.12 Kibana	3	330	June 29, 2021

Kibana cannot perform migrations due to validation error

Related topics