Upgrading Kibana from 7.11 to 7.14

[ankitachow]

I'm trying to upgrade kibana from 7.11 to 7.14. It's on kubernetes. I have kibana deployed with 3 replicas. So after deploying, I can see that 1 pod is up & running whereas the other 2 shows below error.

After deleting .kibana indexes and scaling up again I still see 1 pod down. What's the reason for this?

["error","savedobjects-service"],"pid":14,"message":"[.kibana] Action failed with 'Response Error'. Retrying attempt 3 in 8 seconds.
"error","savedobjects-service"],"pid":14,"message":"[.kibana_task_manager] Action failed with 'Response Error'. Retrying attempt 4 in 16 seconds

[rudolf (Rudolf Meijering)]

Hi,

This message means Elasticsearch returned an HTTP error code with an empty response body. 7.14.1 comes with a fix to provide more detail in these failure logs. In the meantime, if you have a proxy in front of Elasticsearch it would be really useful to see the HTTP response code for this failing request?

It would also be useful to see all the Kibana and Elasticsearch logs during the migration.

I've only seen this happen because Elasticsearch returned a 413 indicating that the payload to Elasticsearch was too large see Migrations should dynamically adjust batch size to prevent failing on 413 errors from Elasticsearch · Issue #107641 · elastic/kibana · GitHub Although this specific error doesn't explain why one of your pods are up but the others aren't, I would expect it to cause all pods to fail.

To reduce the payload size you can try smaller batches by reducing migrations.batchSize to e.g. 500 instead of the default value of 1000.