Upgrade of Kibana from 7.7.1 to 7.8.0

Elastic Stack Kibana
1

I upgraded from Kibana 7.7.1 to 7.8.0 (the elastic search cluster is now running 7.8.0)
When starting Kibana, I get the following error message:

kibana: FATAL Error: Document contains at least one immense term in field="upgrade-assistant-reindex-operation.errorMessage" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '[91, 105, 110, 100, 101, 120, 95, 99, 114, 101, 97, 116, 105, 111, 110, 95, 101, 120, 99, 101, 112, 116, 105, 111, 110, 93, 32, 102, 97, 105]...', original message: bytes can be at most 32766 in length; got 43360

kibana.service: main process exited, code=exited, status=1/FAILURE Unit kibana.service entered failed state.

Then it restarts but will not go any further because:
Another Kibana instance appears to be migrating the index. Waiting for that migration to complete. If no other Kibana instance is attempting migrations, you can get past this message by deleting index .kibana_17 and restarting Kibana."}

How can I fix the original message about the field="upgrade-assistant-reindex-operation.errorMessage ?

Thank you

2

Hi @drewlander,

I'd encourage you to go through https://www.elastic.co/guide/en/kibana/current/upgrade-migrations.html#upgrade-migrations-process and in particular try this instruction. It seems something unexpected has happened during migration so that Kibana stuck in this tate.

Best,
Oleg

3

thank you @azasypkin for the link
I have deleted the referenced .kibana17, and restarted, and I get the same error.
I downgraded kibana, and kibana is up and working.
I went to 8.0 upgrade assistant and it did not have any issues reported.

Is this error message saying that the field "upgrade-assistant-reindex-operation.errorMessage" is not big enough to hold the error message? Is the error message elsewhere? How can I find out what is causing kibana to not like the migration?
Am I missing something important in the documentation you sent?
Thank you

4

anyone?

5

I upgraded two dev cluster and both had problem on one node, didn't restart kibana. other two worked.
something definitely wrong or some kind of bug.

The way I fix is to
save config file
remove rpm.
delete all binary /usr/share/kibana
recreate old uid/gid (same as other cluster node)
reinstall rpm
copy back original config file

6

I'm also getting this error when trying to migrate from Kibana 7.7.1 to 7.8.0. I've tried deleting the new .kibana index and trying again, but the error happens every time. Also tried a new VM with no luck. Has anyone found a solution?

{"type":"log","@timestamp":"2020-07-25T14:45:01Z","tags":["fatal","root"],"pid":15830,"message":"{ Error: Document contains at least one immense term in field=\"upgrade-assistant-reindex-operation.errorMessage\" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped.  Please correct the analyzer to not produce such terms.  The prefix of the first immense term is: '[69, 114, 114, 111, 114, 58, 32, 82, 101, 105, 110, 100, 101, 120, 105, 110, 103, 32, 102, 97, 105, 108, 101, 100, 58, 32, 123, 34, 99, 111]...', original message: bytes can be at most 32766 in length; got 33210\n    at Object.write (/usr/share/kibana/src/core/server/saved_objects/migrations/core/elastic_index.js:152:21)\n    at process._tickCallback (internal/process/next_tick.js:68:7)\n  detail:\n   { index:\n      { _index: '.kibana_10',\n        _type: '_doc',\n        _id:\n         'upgrade-assistant-reindex-operation:1dfe8200-6a2e-11ea-867b-779f318d4c7e',\n        status: 400,\n        error: [Object] } } }"}
7

I followed the workaround in https://github.com/elastic/kibana/issues/71642#issuecomment-658975833 and was able to upgrade successfully to 7.8.0

8

That worked like a charm, thank you!

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.