Everything is broken after upgrade to 7.11

After upgrade to 7.11 everything is broken:
-Elastic-agent in Fleet stopped every agent;
-Stack management won't open;
-Trying to read logs in Discover gives an error:

'Wrapper@https://2e233a9cadac49f5b5e52ee324f8e04f.westeurope.azure.elastic-cloud.com:9243/36136/bundles/core/core.entry.js:6:4249
_createSuperInternal@https://2e233a9cadac49f5b5e52ee324f8e04f.westeurope.azure.elastic-cloud.com:9243/36136/bundles/core/core.entry.js:6:3388
HttpFetchError@https://2e233a9cadac49f5b5e52ee324f8e04f.westeurope.azure.elastic-cloud.com:9243/36136/bundles/core/core.entry.js:6:6016
_callee3$@https://2e233a9cadac49f5b5e52ee324f8e04f.westeurope.azure.elastic-cloud.com:9243/36136/bundles/core/core.entry.js:6:59535
l@https://2e233a9cadac49f5b5e52ee324f8e04f.westeurope.azure.elastic-cloud.com:9243/36136/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:380:982071
s/o._invoke</<@https://2e233a9cadac49f5b5e52ee324f8e04f.westeurope.azure.elastic-cloud.com:9243/36136/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:380:981825
_/</e[t]@https://2e233a9cadac49f5b5e52ee324f8e04f.westeurope.azure.elastic-cloud.com:9243/36136/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:380:982428
fetch_asyncGeneratorStep@https://2e233a9cadac49f5b5e52ee324f8e04f.westeurope.azure.elastic-cloud.com:9243/36136/bundles/core/core.entry.js:6:52652
_next@https://2e233a9cadac49f5b5e52ee324f8e04f.westeurope.azure.elastic-cloud.com:9243/36136/bundles/core/core.entry.js:6:52992'

Hi Herman! Sorry to hear about that.

Can you check if this bug matches your issue? If so, there appears to be a workaround..

Hi Nick,

Thanks for your quick answer.

Do I have to remove the greyed text only, or also everything in between?

Thanks Herman

Hi Nick,

Thank you for your quick reaction. Disregard my previous question. I’ve updated 4 templates and error isn’t occurring anymore.

But now I am getting a new error:

Unable to initialize Fleet

Concurrent installation or upgrade of endpoint-0.17.1 detected, aborting.

Thanks,

Herman

Hi,
Sorry, maybe I was too quick. After a while the first error occurred again and I had to do 6 more template updates.
It's working again!
Thanks,
Herman

I am also unfortunately having this issue. I have run through the workaround, and I also ran into the error:

Unable to initialize Fleet.

Concurrent installation or upgrade of endpoint-0.17.1 detected, aborting.

I have tried refreshing the page multiple times over the past hour or so, looking for more index templates to fix, but I either receive the above error, or I get:

Unable to initialize Fleet

An internal server error occurred.

The workaround does not appear to be working for us.

Hi Eric,

I had the same. But trying sometime later gave me more broken index templates. In the end till 6 or 7 or so.

Fixed them all and problem solved.

Herman

Next day now, and I still cannot get past the internal server error. I did modify several index templates per the workaround, but all I get is the internal server error now.

I do see these in the Kibana logs:

{"type":"log","@timestamp":"2021-02-17T07:35:38-08:00","tags":["error","plugins","fleet"],"pid":806,"message":"Request Timeout after 30000ms"}
{"type":"log","@timestamp":"2021-02-17T07:35:38-08:00","tags":["error","http"],"pid":806,"message":"Error: options.statusCode is expected to be set. given options: undefined
    at Object.customError (/usr/share/kibana/src/core/server/http/router/response.js:136:13)
    at defaultIngestErrorHandler (/usr/share/kibana/x-pack/plugins/fleet/server/errors/handlers.js:117:19)
    at FleetSetupHandler (/usr/share/kibana/x-pack/plugins/fleet/server/routes/setup/handlers.js:111:50)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:163:30)
    at handler (/usr/share/kibana/src/core/server/http/router/router.js:124:50)
    at module.exports.internals.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:45:28)
    at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)
    at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)
    at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:312:32)
    at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:221:9)"}
{"type":"error","@timestamp":"2021-02-17T07:34:58-08:00","tags":[],"pid":806,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error
    at HapiResponseAdapter.toInternalError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:58:19)
    at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:177:34)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at handler (/usr/share/kibana/src/core/server/http/router/router.js:124:50)
    at module.exports.internals.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:45:28)
    at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)
    at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)
    at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:312:32)
    at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:221:9)"},"url":"https://kb.contoso.com/api/fleet/setup","message":"Internal Server Error"}
{"type":"response","@timestamp":"2021-02-17T07:34:58-08:00","tags":["access:fleet-read"],"pid":806,"method":"post","statusCode":500,"req":{"url":"/api/fleet/setup","method":"post","headers":{"host":"kb.contoso.com","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:85.0) Gecko/20100101 Firefox/85.0","accept":"*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br","referer":"https://kb.contoso.com/s/space-name/app/fleet","content-type":"application/json","kbn-version":"7.11.0","origin":"https://kb.contoso.com","connection":"keep-alive","content-length":"0"},"remoteAddress":"192.168.1.100","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:85.0) Gecko/20100101 Firefox/85.0","referer":"https://kb.contoso.com/s/space-name/app/fleet"},"res":{"statusCode":500,"responseTime":40519,"contentLength":9},"message":"POST /api/fleet/setup 500 40519ms - 9.0B"}

I think you are hitting this issue [Fleet] Upgrade to 7.11 failed · Issue #90984 · elastic/kibana · GitHub . It will be shipped with 7.11.1

Thanks
Rashmi

It is the same issue, but the workaround that seems to be working for others did not work for me.

@MakoWish I think you might be seeing the same problem as described in Unable to initialize Fleet - An internal server error occurred

We're tracking this in [Fleet] [Security Solution] Fleet cannot upgrade: Transform stop times out · Issue #91570 · elastic/kibana · GitHub

7.11.1 did not resolve the issue.

It made matters worse not sure if @MakoWish has had any luck but I for one ran into his issue with 7.11. 7.11.1 has another error where it will not connect to Elastic at all when it is clearly running and accepting connections.

7.11.1 also did not fix the issue for me. I am still receiving the same error.

Unable to initialize Fleet

An internal server error occurred.

Adding the transform node solved my issue!

1 Like

I just saw that this morning and came here to link to the post, but you beat me to it! :slight_smile: Adding the Transform role solved the issue for me as well.

Slightly off-topic, but question on the role... I enabled it on both our Kibana servers for now (ES coordinating nodes), but what node does "best practice" say to enable it on?