Kibana cannot perform migrations due to validation error

Kibana is unable to perform migrations and then reports it cannot connect to elastic search.
elasticsearch is running locally.

Upgrading to 7.11.2 elasticsearch and kibana.

"type":"log","@timestamp":"2021-07-14T14:02:53+00:00","tags":["info","savedobjects-service"],"pid":830523,"message":"Starting saved objects migrations"}
{"type":"log","@timestamp":"2021-07-14T14:02:53+00:00","tags":["info","savedobjects-service"],"pid":830523,"message":"Creating index .kibana_task_manager_1."}
{"type":"log","@timestamp":"2021-07-14T14:02:53+00:00","tags":["warning","savedobjects-service"],"pid":830523,"message":"Unable to connect to Elasticsearch. Error: validation_exception"}
{"type":"log","@timestamp":"2021-07-14T14:02:53+00:00","tags":["fatal","root"],"pid":830523,"message":"ResponseError: validation_exception\n    at onBody (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:333:23)\n    at IncomingMessage.onEnd (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:260:11)\n    at IncomingMessage.emit (events.js:327:22)\n    at endReadableNT (internal/streams/readable.js:1327:12)\n    at processTicksAndRejections (internal/process/task_queues.js:80:21) {\n  meta: {\n    body: { error: [Object], status: 400 },\n    statusCode: 400,\n    headers: {\n      'content-type': 'application/json; charset=UTF-8',\n      'content-length': '379'\n    },\n    meta: {\n      context: null,\n      request: [Object],\n      name: 'elasticsearch-js',\n      connection: [Object],\n      attempts: 0,\n      aborted: false\n    }\n  }\n}"}
{"type":"log","@timestamp":"2021-07-14T14:02:53+00:00","tags":["info","plugins-system"],"pid":830523,"message":"Stopping all plugins."}
{"type":"log","@timestamp":"2021-07-14T14:02:53+00:00","tags":["info","plugins","monitoring","monitoring","kibana-monitoring"],"pid":830523,"message":"Monitoring stats collection is stopped"}
{"type":"log","@timestamp":"2021-07-14T14:02:53+00:00","tags":["info","savedobjects-service"],"pid":830523,"message":"Creating index .kibana_2."}
{"type":"log","@timestamp":"2021-07-14T14:02:53+00:00","tags":["warning","savedobjects-service"],"pid":830523,"message":"Unable to connect to Elasticsearch. Error: validation_exception"}

@dbernard please see this discuss post that walks through some debugging steps. If that doesn't solve your issue, could you provide more verbose logs please?

There are a bunch of start up errors I'll look at.
kibana 6.2.4-1 installation was inadvertently upgraded to 7.10 then 7.12 by accident.

{"type":"log","@timestamp":"2021-07-15T00:19:26+00:00","tags":["warning","plugins","licensing"],"pid":1837423,"message":"License information could not be obtained from Elasticsearch due to Error: Cluster client cannot be used after it has been closed. error"}
{"type":"log","@timestamp":"2021-07-15T00:19:56+00:00","tags":["warning","plugins","licensing"],"pid":1837423,"message":"License information could not be obtained from Elasticsearch due to Error: Cluster client cannot be used after it has been closed. error"}
{"type":"log","@timestamp":"2021-07-15T00:20:26+00:00","tags":["warning","plugins","licensing"],"pid":1837423,"message":"License information could not be obtained from Elasticsearch due to Error: Cluster client cannot be used after it has been closed. error"}
{"type":"log","@timestamp":"2021-07-15T00:20:56+00:00","tags":["warning","plugins","licensing"],"pid":1837423,"message":"License information could not be obtained from Elasticsearch due to Error: Cluster client cannot be used after it has been closed. error"}
{"type":"log","@timestamp":"2021-07-15T00:21:10+00:00","tags":["info","plugins-system"],"pid":1837423,"message":"Stopping all plugins."}
{"type":"log","@timestamp":"2021-07-15T00:21:17+00:00","tags":["info","plugins-service"],"pid":1845073,"message":"Plugin \"visTypeXy\" is disabled."}
{"type":"log","@timestamp":"2021-07-15T00:21:17+00:00","tags":["warning","config","deprecation"],"pid":1845073,"message":"Setting [elasticsearch.username] to \"elastic\" is deprecated. You should use the \"kibana_system\" user instead."}
{"type":"log","@timestamp":"2021-07-15T00:21:17+00:00","tags":["warning","config","deprecation"],"pid":1845073,"message":"Config key [monitoring.cluster_alerts.email_notifications.email_address] will be required for email notifications to work in 8.0.\""}
{"type":"log","@timestamp":"2021-07-15T00:21:17+00:00","tags":["warning","config","deprecation"],"pid":1845073,"message":"Setting [monitoring.username] to \"elastic\" is deprecated. You should use the \"kibana_system\" user instead."}
{"type":"log","@timestamp":"2021-07-15T00:21:18+00:00","tags":["info","plugins-system"],"pid":1845073,"message":"Setting up [102] plugins: [taskManager,licensing,globalSearch,globalSearchProviders,code,usageCollection,xpackLegacy,telemetryCollectionManager,telemetry,telemetryCollectionXpack,kibanaUsageCollection,securityOss,newsfeed,mapsLegacy,kibanaLegacy,translations,share,legacyExport,embeddable,uiActionsEnhanced,expressions,charts,esUiShared,bfetch,data,home,observability,console,consoleExtensions,apmOss,searchprofiler,painlessLab,grokdebugger,management,indexPatternManagement,advancedSettings,fileUpload,savedObjects,visualizations,visTypeVislib,visTypeVega,visTypeTimeseries,visTypeTimeseriesEnhanced,visTypeTimelion,features,licenseManagement,dataEnhanced,watcher,canvas,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeMarkdown,tileMap,regionMap,mapsOss,lensOss,inputControlVis,graph,timelion,dashboard,dashboardEnhanced,visualize,discover,discoverEnhanced,savedObjectsManagement,spaces,security,savedObjectsTagging,maps,lens,reporting,lists,encryptedSavedObjects,dashboardMode,cloud,upgradeAssistant,snapshotRestore,fleet,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,enterpriseSearch,ml,beatsManagement,wazuh,transform,ingestPipelines,eventLog,actions,alerts,triggersActionsUi,stackAlerts,securitySolution,case,infra,monitoring,logstash,apm,uptime]"}
{"type":"log","@timestamp":"2021-07-15T00:21:18+00:00","tags":["info","plugins","taskManager"],"pid":1845073,"message":"TaskManager is identified by the Kibana UUID: d288334a-6d1f-4f72-ae5d-145112a02759"}
{"type":"log","@timestamp":"2021-07-15T00:21:18+00:00","tags":["warning","plugins","security","config"],"pid":1845073,"message":"Session cookies will be transmitted over insecure connections. This is not recommended."}
{"type":"log","@timestamp":"2021-07-15T00:21:18+00:00","tags":["warning","plugins","reporting","config"],"pid":1845073,"message":"Chromium sandbox provides an additional layer of protection, but is not supported for Linux CentOS 7.9.2009 OS. Automatically setting 'xpack.reporting.capture.browser.chromium.disableSandbox: true'."}
{"type":"log","@timestamp":"2021-07-15T00:21:18+00:00","tags":["info","plugins","monitoring","monitoring"],"pid":1845073,"message":"config sourced from: production cluster"}
{"type":"log","@timestamp":"2021-07-15T00:21:18+00:00","tags":["info","savedobjects-service"],"pid":1845073,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
{"type":"log","@timestamp":"2021-07-15T00:21:19+00:00","tags":["info","savedobjects-service"],"pid":1845073,"message":"Starting saved objects migrations"}
{"type":"log","@timestamp":"2021-07-15T00:21:19+00:00","tags":["info","savedobjects-service"],"pid":1845073,"message":"Creating index .kibana_task_manager_1."}
{"type":"log","@timestamp":"2021-07-15T00:21:19+00:00","tags":["warning","savedobjects-service"],"pid":1845073,"message":"Unable to connect to Elasticsearch. Error: validation_exception"}
{"type":"log","@timestamp":"2021-07-15T00:21:19+00:00","tags":["fatal","root"],"pid":1845073,"message":"ResponseError: validation_exception\n    at onBody (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:333:23)\n    at IncomingMessage.onEnd (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:260:11)\n    at IncomingMessage.emit (events.js:327:22)\n    at endReadableNT (internal/streams/readable.js:1327:12)\n    at processTicksAndRejections (internal/process/task_queues.js:80:21) {\n  meta: {\n    body: { error: [Object], status: 400 },\n    statusCode: 400,\n    headers: {\n      'content-type': 'application/json; charset=UTF-8',\n      'content-length': '379'\n    },\n    meta: {\n      context: null,\n      request: [Object],\n      name: 'elasticsearch-js',\n      connection: [Object],\n      attempts: 0,\n      aborted: false\n    }\n  }\n}"}
{"type":"log","@timestamp":"2021-07-15T00:21:19+00:00","tags":["info","plugins-system"],"pid":1845073,"message":"Stopping all plugins."}
{"type":"log","@timestamp":"2021-07-15T00:21:19+00:00","tags":["info","plugins","monitoring","monitoring","kibana-monitoring"],"pid":1845073,"message":"Monitoring stats collection is stopped"}
{"type":"log","@timestamp":"2021-07-15T00:21:19+00:00","tags":["info","savedobjects-service"],"pid":1845073,"message":"Creating index .kibana_1."}
{"type":"log","@timestamp":"2021-07-15T00:21:19+00:00","tags":["warning","savedobjects-service"],"pid":1845073,"message":"Unable to connect to Elasticsearch. Error: validation_exception"}
{"type":"log","@timestamp":"2021-07-15T00:21:48+00:00","tags":["warning","plugins","licensing"],"pid":1845073,"message":"License information could not be obtained from Elasticsearch due to Error: Cluster client cannot be used after it has been closed. error"}
{"type":"log","@timestamp":"2021-07-15T00:22:18+00:00","tags":["warning","plugins","licensing"],"pid":1845073,"message":"License information could not be obtained from Elasticsearch due to Error: Cluster client cannot be used after it has been closed. error"}
{"type":"log","@timestamp":"2021-07-15T00:22:48+00:00","tags":["warning","plugins","licensing"],"pid":1845073,"message":"License information could not be obtained from Elasticsearch due to Error: Cluster client cannot be u

kibana.yml is minimal

server.port: 5601
server.name: "elk101"
elasticsearch.hosts: [ "http://localhost:9200" ]
elasticsearch.requestTimeout: 3000000 

server.host: "0.0.0.0"

xpack.security.enabled: true
elasticsearch.username: elastic
elasticsearch.password: changeme

uiSettings.overrides.defaultRoute: "/app/wazuh"
elasticsearch.ssl.verificationMode: none

Looks like the problem was with the upgraded elasticsearch not having a large enough maximum shards per node.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.