Everything is broken after upgrade to 7.11

hermlam · February 11, 2021, 9:21am

After upgrade to 7.11 everything is broken:
-Elastic-agent in Fleet stopped every agent;
-Stack management won't open;
-Trying to read logs in Discover gives an error:

nickpeihl · February 11, 2021, 11:45pm

Hi Herman! Sorry to hear about that.

Can you check if this bug matches your issue? If so, there appears to be a workaround..

hermlam · February 12, 2021, 10:33am

Hi Nick,

Thanks for your quick answer.

Do I have to remove the greyed text only, or also everything in between?

Thanks Herman

hermlam · February 12, 2021, 10:47am

Hi Nick,

Thank you for your quick reaction. Disregard my previous question. I’ve updated 4 templates and error isn’t occurring anymore.

But now I am getting a new error:

Unable to initialize Fleet

Concurrent installation or upgrade of endpoint-0.17.1 detected, aborting.

Thanks,

Herman

hermlam · February 12, 2021, 11:03am

Hi,
Sorry, maybe I was too quick. After a while the first error occurred again and I had to do 6 more template updates.
It's working again!
Thanks,
Herman

MakoWish · February 16, 2021, 10:54pm

I am also unfortunately having this issue. I have run through the workaround, and I also ran into the error:

Unable to initialize Fleet.

Concurrent installation or upgrade of endpoint-0.17.1 detected, aborting.

I have tried refreshing the page multiple times over the past hour or so, looking for more index templates to fix, but I either receive the above error, or I get:

Unable to initialize Fleet

An internal server error occurred.

The workaround does not appear to be working for us.

hermlam · February 17, 2021, 1:46am

Hi Eric,

I had the same. But trying sometime later gave me more broken index templates. In the end till 6 or 7 or so.

Fixed them all and problem solved.

Herman

MakoWish · February 17, 2021, 3:43pm

Next day now, and I still cannot get past the internal server error. I did modify several index templates per the workaround, but all I get is the internal server error now.

I do see these in the Kibana logs:

{"type":"log","@timestamp":"2021-02-17T07:35:38-08:00","tags":["error","plugins","fleet"],"pid":806,"message":"Request Timeout after 30000ms"}
{"type":"log","@timestamp":"2021-02-17T07:35:38-08:00","tags":["error","http"],"pid":806,"message":"Error: options.statusCode is expected to be set. given options: undefined
    at Object.customError (/usr/share/kibana/src/core/server/http/router/response.js:136:13)
    at defaultIngestErrorHandler (/usr/share/kibana/x-pack/plugins/fleet/server/errors/handlers.js:117:19)
    at FleetSetupHandler (/usr/share/kibana/x-pack/plugins/fleet/server/routes/setup/handlers.js:111:50)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:163:30)
    at handler (/usr/share/kibana/src/core/server/http/router/router.js:124:50)
    at module.exports.internals.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:45:28)
    at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)
    at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)
    at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:312:32)
    at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:221:9)"}
{"type":"error","@timestamp":"2021-02-17T07:34:58-08:00","tags":[],"pid":806,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error
    at HapiResponseAdapter.toInternalError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:58:19)
    at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:177:34)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at handler (/usr/share/kibana/src/core/server/http/router/router.js:124:50)
    at module.exports.internals.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:45:28)
    at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)
    at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)
    at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:312:32)
    at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:221:9)"},"url":"https://kb.contoso.com/api/fleet/setup","message":"Internal Server Error"}
{"type":"response","@timestamp":"2021-02-17T07:34:58-08:00","tags":["access:fleet-read"],"pid":806,"method":"post","statusCode":500,"req":{"url":"/api/fleet/setup","method":"post","headers":{"host":"kb.contoso.com","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:85.0) Gecko/20100101 Firefox/85.0","accept":"*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br","referer":"https://kb.contoso.com/s/space-name/app/fleet","content-type":"application/json","kbn-version":"7.11.0","origin":"https://kb.contoso.com","connection":"keep-alive","content-length":"0"},"remoteAddress":"192.168.1.100","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:85.0) Gecko/20100101 Firefox/85.0","referer":"https://kb.contoso.com/s/space-name/app/fleet"},"res":{"statusCode":500,"responseTime":40519,"contentLength":9},"message":"POST /api/fleet/setup 500 40519ms - 9.0B"}

rashmi · February 17, 2021, 3:58pm

I think you are hitting this issue [Fleet] Upgrade to 7.11 failed · Issue #90984 · elastic/kibana · GitHub . It will be shipped with 7.11.1

Thanks
Rashmi

MakoWish · February 17, 2021, 4:00pm

It is the same issue, but the workaround that seems to be working for others did not work for me.

skh · February 17, 2021, 4:16pm

@MakoWish I think you might be seeing the same problem as described in Unable to initialize Fleet - An internal server error occurred

We're tracking this in [Fleet] [Security Solution] Fleet cannot upgrade: Transform stop times out · Issue #91570 · elastic/kibana · GitHub

PublicName · February 17, 2021, 8:20pm

7.11.1 did not resolve the issue.

It made matters worse not sure if @MakoWish has had any luck but I for one ran into his issue with 7.11. 7.11.1 has another error where it will not connect to Elastic at all when it is clearly running and accepting connections.

MakoWish · February 17, 2021, 9:37pm

7.11.1 also did not fix the issue for me. I am still receiving the same error.

Unable to initialize Fleet

An internal server error occurred.

searchtool · February 17, 2021, 9:38pm

Adding the transform node solved my issue!

MakoWish · February 18, 2021, 4:17pm

I just saw that this morning and came here to link to the post, but you beat me to it! Adding the Transform role solved the issue for me as well.

Slightly off-topic, but question on the role... I enabled it on both our Kibana servers for now (ES coordinating nodes), but what node does "best practice" say to enable it on?

system · March 18, 2021, 4:17pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.