Kibana Plugin Development Issue on AWS ec2

Hello,

I am running the development environment on an AWS ec2 instance with 8GB of RAM, 25GB disk space and on AWS Linux 2. I have read somewhere that it this environment has issues running on the web.

I was following the Getting started guide here : https://www.elastic.co/guide/en/kibana/current/development-getting-started.html and ran into an issue at the last step in when I go to run Kibana.

yarn start --run-examples

It give the following error:

Unhandled Promise rejection detected:

Error: Client network socket disconnected before secure TLS connection was established

I have the following trace snippet:
NOTE: I've replaced some of the text to be under the max. limit of characters for this message.

[ec2-user@ip-172-31-12-6 kibana]$ yarn start --run-examples
yarn run v1.22.10
$ node scripts/kibana --dev --run-examples
[2020-12-13T14:46:33.701Z][INFO ][plugins-service] Plugin initialization disabled.
[2020-12-13T14:46:33.770Z][WARN ][savedobjects-service] Skipping Saved Object migrations on startup. Note: Individual documents will still be migrated when read or written.
 no-base-path  ====================================================================================================
 no-base-path  Running Kibana in dev mode with --no-base-path disables several useful features and is not recommended
 no-base-path  ====================================================================================================
np bld    log   [14:46:35.250] [info][@kbn/optimizer] initialized, 32 bundles cached
np bld    log   [14:46:35.257] [warning][@kbn/optimizer] only building [v7dark,v7light] themes, customize with the KBN_OPTIMIZER_THEMES environment variable
np bld    log   [14:46:37.876] [info][@kbn/optimizer] starting worker [43 bundles]
np bld    log   [14:46:37.892] [info][@kbn/optimizer] starting worker [45 bundles]
 watching for changes  (7954 files)
 server    log   [14:47:03.030] [info][plugins-service] Plugin "visTypeXy" is disabled.
 server    log   [14:47:04.600] [info][plugins-system] Setting up [107] plugins: [...]
 server    log   [14:47:04.630] [info][plugins][taskManager] TaskManager is identified by the Kibana UUID: 64f54d40-ee3e-499b-9e95-f5dfe88ec3d8
 server    log   [14:47:05.575] [warning][config][plugins][security] Session cookies will be transmitted over insecure connections. This is not recommended.
 server    log   [14:47:05.786] [info][config][plugins][reporting] Chromium sandbox provides an additional layer of protection, and is supported for Linux Amazon Linux 2 (Karoo) OS. Automatically enabling Chromium sandbox.
 server    log   [14:47:06.741] [info][monitoring][monitoring][plugins] config sourced from: production cluster
 server    log   [14:47:07.926] [info][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...
 server    log   [14:47:08.233] [info][savedobjects-service] Starting saved objects migrations
 server    log   [14:47:08.931] [info][plugins-system] Starting [107] plugins: [...]
 [BABEL] Note: The code generator has deoptimised the styling of /home/ec2-user/repos/kibana/x-pack/plugins/canvas/server/templates/pitch_presentation.ts as it exceeds the max of 500KB.
 server    log   [14:47:17.847] [info][plugins][reporting] Downloading https://storage.googleapis.com/headless_shell/chromium-ef768c9-darwin_x64.zip to /home/ec2-user/repos/kibana/.chromium/chromium-ef768c9-darwin_x64.zip
 server    log   [14:47:18.020] [info][plugins][reporting] Downloading https://storage.googleapis.com/headless_shell/chromium-ef768c9-linux_x64.zip to /home/ec2-user/repos/kibana/.chromium/chromium-ef768c9-linux_x64.zip
 server    log   [14:47:18.048] [info][plugins][reporting] Downloading https://storage.googleapis.com/headless_shell/chromium-ef768c9-linux_arm64.zip to /home/ec2-user/repos/kibana/.chromium/chromium-ef768c9-linux_arm64.zip
 server    log   [14:47:18.070] [info][plugins][reporting] Downloading https://storage.googleapis.com/headless_shell/chromium-ef768c9-windows_x64.zip to /home/ec2-user/repos/kibana/.chromium/chromium-ef768c9-windows_x64.zip
 [BABEL] Note: The code generator has deoptimised the styling of /home/ec2-user/repos/kibana/x-pack/plugins/security_solution/server/utils/beat_schema/fields.ts as it exceeds the max of 500KB.
 server    log   [14:47:43.955] [info][listening] Server running at http://localhost:5601
 server    log   [14:47:49.515] [warning][licensing][plugins] License information could not be obtained from Elasticsearch due to Error: Request Timeout after 30000ms error
 server    log   [14:47:49.521] [info][plugins][searchprofiler] You cannot use searchprofiler because license information is not available at this time.
 server    log   [14:47:49.527] [info][painlessLab][plugins] You cannot use painlessLab because license information is not available at this time.
 server    log   [14:47:49.537] [info][plugins][watcher] You cannot use watcher because license information is not available at this time.
 server    log   [14:47:49.560] [info][plugins][snapshotRestore] You cannot use snapshot_restore because license information is not available at this time.
 server    log   [14:47:49.561] [info][indexManagement][plugins] You cannot use index_management because license information is not available at this time.
 server    log   [14:47:49.561] [info][plugins][rollup] You cannot use rollup because license information is not available at this time.
 server    log   [14:47:49.561] [info][plugins][remoteClusters] You cannot use Remote Clusters because license information is not available at this time.
 server    log   [14:47:49.561] [info][crossClusterReplication][plugins] You cannot use crossClusterReplication because license information is not available at this time.
 server    log   [14:47:49.562] [info][indexLifecycleManagement][plugins] You cannot use index_lifecycle_management because license information is not available at this time.
 server    log   [14:47:49.562] [info][plugins][transform] You cannot use transform because license information is not available at this time.
 server    log   [14:47:49.562] [info][ingestPipelines][plugins] You cannot use ingest_pipelines because license information is not available at this time.
 server    log   [14:47:49.575] [info][kibana-monitoring][monitoring][monitoring][plugins] Monitoring status upload endpoint is not enabled in Elasticsearch:Monitoring stats collection is stopped
 server    log   [14:47:49.721] [error][plugins][taskManager] Failed to poll for work: Error: work has timed out
 server    log   [14:47:49.736] [warning][plugins][telemetry] Unable to update legacy telemetry configs.
 server    log   [14:47:49.747] [error][ml][plugins] Error Initializing jobs {"name":"TimeoutError","meta":{"body":null,"statusCode":null,"headers":null,"meta":{"context":null,"request":{"params":{"method":"POST","path":"/.kibana/_search","body":"{\"seq_no_primary_term\":true,\"query\":{\"bool\":{\"filter\":[{\"bool\":{\"should\":[{\"bool\":{\"must\":[{\"term\":{\"type\":\"ml-job\"}}],\"must_not\":[{\"exists\":{\"field\":\"namespace\"}}]}}],\"minimum_should_match\":1}}]}}}","querystring":"size=0&from=0&rest_total_hits_as_int=true","headers":{"user-agent":"elasticsearch-js/7.10.0 (linux 4.14.209-160.335.amzn2.x86_64-x64; Node.js v14.15.1)","x-elastic-product-origin":"kibana","content-type":"application/json","content-length":"198"},"timeout":30000},"options":{"maxRetries":0,"ignore":[404]},"id":15},"name":"elasticsearch-js","connection":{"url":"http://localhost:9200/","id":"http://localhost:9200/","headers":{},"deadCount":5,"resurrectTimeout":1607871829747,"_openRequests":13,"status":"dead","roles":{"master":true,"data":true,"ingest":true,"ml":false}},"attempts":0,"aborted":false}},"isBoom":true,"isServer":true,"data":null,"output":{"statusCode":503,"payload":{"statusCode":503,"error":"Service Unavailable","message":"Request timed out"},"headers":{}}}
 server    log   [14:47:49.796] [info][server][Kibana][http] http server running at http://localhost:5601
 Unhandled Promise rejection detected:

 Error: Client network socket disconnected before secure TLS connection was established
 at connResetException (internal/errors.js:617:14)
 at TLSSocket.onConnectEnd (_tls_wrap.js:1544:19)
 at TLSSocket.emit (events.js:327:22)
 at endReadableNT (_stream_readable.js:1327:12)
 at processTicksAndRejections (internal/process/task_queues.js:80:21) {
 code: 'ECONNRESET',
 path: null,
 host: 'storage.googleapis.com',
 port: 443,
 localAddress: undefined,
 config: {
 url: 'https://storage.googleapis.com/headless_shell/chromium-ef768c9-darwin_x64.zip',
 method: 'get',
 headers: {
 Accept: 'application/json, text/plain, */*',
 'User-Agent': 'axios/0.19.2'
 },
 transformRequest: [ [Function: transformRequest] ],
 transformResponse: [ [Function: transformResponse] ],
 timeout: 0,
 adapter: [Function: httpAdapter],
 responseType: 'stream',
 xsrfCookieName: 'XSRF-TOKEN',
 xsrfHeaderName: 'X-XSRF-TOKEN',
 maxContentLength: -1,
 validateStatus: [Function: validateStatus],
 data: undefined
 },
 request: <ref *1> Writable {
 _writableState: WritableState {
 objectMode: false,
 highWaterMark: 16384,
 finalCalled: false,
 needDrain: false,
 ending: false,
 ended: false,
 finished: false,
 destroyed: false,
 decodeStrings: true,
 defaultEncoding: 'utf8',
 length: 0,
 writing: false,
 corked: 0,
 sync: true,
 bufferProcessing: false,
 onwrite: [Function: bound onwrite],
 writecb: null,
 writelen: 0,
 afterWriteTickInfo: null,
 buffered: [],
 bufferedIndex: 0,
 allBuffers: true,
 allNoop: true,
 pendingcb: 0,
 prefinished: false,
 errorEmitted: false,
 emitClose: true,
 autoDestroy: true,
 errored: null,
 closed: false
 },
 _events: [Object: null prototype] {
 response: [Function: handleResponse],
 error: [Function: handleRequestError]
 },
 _eventsCount: 2,
 _maxListeners: undefined,
 _options: {
 protocol: 'https:',
 maxRedirects: 21,
 maxBodyLength: 10485760,
 path: '/headless_shell/chromium-ef768c9-darwin_x64.zip',
 method: 'GET',
 headers: [Object],
 agent: undefined,
 agents: [Object],
 auth: undefined,
 hostname: 'storage.googleapis.com',
 port: null,
 nativeProtocols: [Object],
 pathname: '/headless_shell/chromium-ef768c9-darwin_x64.zip'
 },
 _redirectCount: 0,
 _redirects: [],
 _requestBodyLength: 0,
 _requestBodyBuffers: [],
 _onNativeResponse: [Function (anonymous)],
 _currentRequest: ClientRequest {
 _events: [Object: null prototype],
 _eventsCount: 6,
 _maxListeners: undefined,
 outputData: [],
 outputSize: 0,
 writable: true,
 destroyed: false,
 _last: true,
 chunkedEncoding: false,
 shouldKeepAlive: false,
 _defaultKeepAlive: true,
 useChunkedEncodingByDefault: false,
 sendDate: false,
 _removedConnection: false,
 _removedContLen: false,
 _removedTE: false,
 _contentLength: 0,
 _hasBody: true,
 _trailer: '',
 finished: true,
 _headerSent: true,
 socket: [TLSSocket],
 _header: 'GET /headless_shell/chromium-ef768c9-darwin_x64.zip HTTP/1.1\r\n' +
 'Accept: application/json, text/plain, */*\r\n' +
 'User-Agent: axios/0.19.2\r\n' +
 'Host: storage.googleapis.com\r\n' +
 'Connection: close\r\n' +
 '\r\n',
 _keepAliveTimeout: 0,
 _onPendingData: [Function: noopPendingOutput],
 agent: [Agent],
 socketPath: undefined,
 method: 'GET',
 maxHeaderSize: undefined,
 insecureHTTPParser: undefined,
 path: '/headless_shell/chromium-ef768c9-darwin_x64.zip',
 _ended: false,
 res: null,
 aborted: false,
 timeoutCb: null,
 upgradeOrConnect: false,
 parser: null,
 maxHeadersCount: null,
 reusedSocket: false,
 host: 'storage.googleapis.com',
 protocol: 'https:',
 _redirectable: [Circular *1],
 [Symbol(kCapture)]: false,
 [Symbol(kNeedDrain)]: false,
 [Symbol(corked)]: 0,
 [Symbol(kOutHeaders)]: [Object: null prototype]
 },
 _currentUrl: 'https://storage.googleapis.com/headless_shell/chromium-ef768c9-darwin_x64.zip',
 [Symbol(kCapture)]: false
 },
 response: undefined,
 isAxiosError: true,
 toJSON: [Function (anonymous)]
 }

 Terminating process...
 server crashed  with status code 1

PS: I did get some warnings at the bootstrap step, but there was no failure at this point.

yarn kbn bootstrap

Does anyone has any insights? Any suggestions? Has anyone set up a docker image?

Thanks for you help,
Mike

I'm curious why you want to run a dev version of Kibana on AWS ec2. Normally you'd run an already -built, production version. We have various deployment stories for that.

The error you're seeing looks like it's trying to download a Mac version of the chrome headless shell, presumably for reporting. You could try disabling the reporting plugin, by using the following in your kibana.yml:

xpack.reporting.enabled: false

Beyond that, not clear why it can't download the file in the first place. Is the networking locked down on this machine?

Hello,

Thanks for your reply, but I'm trying to create a plugin for Kibana and following the Getting started guide here : https://www.elastic.co/guide/en/kibana/current/development-getting-started.html.

I would like to set up this dev environment on an AWS EC2 instance, if possible.

The instance has all the ports open incoming and outgoing.

Mike

I'm not aware of any one else running Kibana in dev mode on an ec2 instance (or any kind of cloud hosting story). Typically we develop plugins on our local machines. I'd be kind of afraid there may be other issues doing this. But I'll ask around, internally.

I could see setting up an ec2 instance (or docker image) with a "production" Kibana that also includes your plugin, but I'm not quite sure why you would want to do all the development in the ec2 instance - you're editing your plugin code there as well? I assume you have an accessible elasticsearch instance that this Kibana will connect to as well?

You mentioned warnings during bootstrap - could you post those here?

Could you try to curl that url that's failing, from the command line, while on your ec2 instance? Looks like it's:

 https://storage.googleapis.com/headless_shell/chromium-ef768c9-darwin_x64.zip

Did you try disabling reporting yet, to see if that will disable the download that's failing?

Heard back from one developer:

I ran kibana 7.8 in development mode on ec2 without any issues multiple times. i think they need to configure their outbound to allow the server to access googleapis for downloading chromium

Hello,

I could see setting up an ec2 instance (or docker image) with a "production" Kibana that also includes your plugin, but I'm not quite sure why you would want to do all the development in the ec2 instance - you're editing your plugin code there as well? I assume you have an accessible elasticsearch instance that this Kibana will connect to as well?

I would like to perform the development (editing, testing, etc...) on the ec2 instance and I have a Windows machine with little resources to work upon. I would have Elasticsearch on the instance as well.

I will perform the Starting up steps again and post the other bootstrap issue soon. They were related to version misalignment of a few packages.

I will try the curl as well and report back.

Mike

1) xpack.reporting.enabled: false

[ec2-user@ip-172-31-24-126 kibana] yarn start --run-examples yarn run v1.22.10 node scripts/kibana --dev --run-examples
[2020-12-16T21:43:30.931Z][INFO ][plugins-service] Plugin initialization disabled.
[2020-12-16T21:43:30.994Z][WARN ][savedobjects-service] Skipping Saved Object migrations on startup. Note: Individual documents will still be migrated when read or written.
no-base-path ====================================================================================================
no-base-path Running Kibana in dev mode with --no-base-path disables several useful features and is not recommended
no-base-path ====================================================================================================
np bld log [21:43:32.478] [info][@kbn/optimizer] initialized, 15 bundles cached
np bld log [21:43:32.484] [warning][@kbn/optimizer] only building [v7dark,v7light] themes, customize with the KBN_OPTIMIZER_THEMES environment variable
np bld log [21:43:34.960] [info][@kbn/optimizer] starting worker [53 bundles]
np bld log [21:43:34.966] [info][@kbn/optimizer] starting worker [52 bundles]
watching for changes (8015 files)

At this point, it sits here.

I don't see it connect to the elasticsearch database, well there is no info log console update saying Kibana is connecting (maybe there isn't a log outpit?).

I tried to navigate to the instance_ip:5601 but the connection is refused.

2) Could you try to curl that url that's failing, from the command line, while on your ec2 instance? Looks like it's:

 [https://storage.googleapis.com/headless_shell/chromium-ef768c9-darwin_x64.zip](https://storage.googleapis.com/headless_shell/chromium-ef768c9-darwin_x64.zip)

Did you mean wget?
[ec2-user@ip-172-31-24-126 ~]$ wget https://storage.googleapis.com/headless_shell/chromium-ef768c9-darwin_x64.zip
--2020-12-16 21:15:03-- https://storage.googleapis.com/headless_shell/chromium-ef768c9-darwin_x64.zip
Resolving storage.googleapis.com (storage.googleapis.com)... 216.58.194.176, 172.217.6.80, 142.250.72.208, ...
Connecting to storage.googleapis.com (storage.googleapis.com)|216.58.194.176|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 72439722 (69M) [application/zip]
Saving to: ‘chromium-ef768c9-darwin_x64.zip’

100%[===================================================================================================================================================================================================================================================================================>] 72,439,722 59.3MB/s in 1.2s

2020-12-16 21:15:05 (59.3 MB/s) - ‘chromium-ef768c9-darwin_x64.zip’ saved [72439722/72439722]

3) You mentioned warnings during bootstrap - could you post those here?
I have pulled the latest version as of today 16-DEC-2020 @ 4:20PM EST:

[ec2-user@ip-172-31-24-126 kibana] cat .node-version 14.15.2 [ec2-user@ip-172-31-24-126 kibana] nvm use 14.15.2
N/A: version "14.15.2 -> N/A" is not yet installed.

You need to run "nvm install 14.15.2" to install it before using it.
[ec2-user@ip-172-31-24-126 kibana] nvm install 14.15.2 Downloading and installing node v14.15.2... Downloading [https://nodejs.org/dist/v14.15.2/node-v14.15.2-linux-x64.tar.xz.](https://nodejs.org/dist/v14.15.2/node-v14.15.2-linux-x64.tar.xz.).. ######################################################################################################################################################################################################################################################### 100.0% Computing checksum with sha256sum Checksums matched! Now using node v14.15.2 (npm v6.14.9) [ec2-user@ip-172-31-24-126 kibana] npm install -g yarn

yarn@1.22.10 preinstall /home/ec2-user/.nvm/versions/node/v14.15.2/lib/node_modules/yarn
:; (node ./preinstall.js > /dev/null 2>&1 || true)

/home/ec2-user/.nvm/versions/node/v14.15.2/bin/yarn -> /home/ec2-user/.nvm/versions/node/v14.15.2/lib/node_modules/yarn/bin/yarn.js
/home/ec2-user/.nvm/versions/node/v14.15.2/bin/yarnpkg -> /home/ec2-user/.nvm/versions/node/v14.15.2/lib/node_modules/yarn/bin/yarn.js

  • yarn@1.22.10
    added 1 package in 1.669s
    [ec2-user@ip-172-31-24-126 kibana] yarn kbn bootstrap yarn run v1.22.10 node scripts/kbn bootstrap
    info [kibana] running yarn

$ node ./preinstall_check
[1/5] Validating package.json...
[2/5] Resolving packages...
warning Resolution field "trim@0.0.3" is incompatible with requested version "trim@0.0.1"
warning Resolution field "trim@0.0.3" is incompatible with requested version "trim@0.0.1"
warning Resolution field "@types/node@14.14.14" is incompatible with requested version "@types/node@^12.0.2"
warning Resolution field "@types/node@14.14.14" is incompatible with requested version "@types/node@8.10.54"
warning Resolution field "typescript@4.1.2" is incompatible with requested version "typescript@~3.7.2"
warning Resolution field "typescript@4.1.2" is incompatible with requested version "typescript@^3.5.3"
warning Resolution field "typescript@4.1.2" is incompatible with requested version "typescript@^3.2.2"
warning Resolution field "schema-utils@1.0.0" is incompatible with requested version "schema-utils@^0.3.0"
warning Resolution field "trim@0.0.3" is incompatible with requested version "trim@0.0.1"
warning Resolution field "trim@0.0.3" is incompatible with requested version "trim@0.0.1"
warning Resolution field "typescript@4.1.2" is incompatible with requested version "typescript@^3.3.3333"
warning Resolution field "trim@0.0.3" is incompatible with requested version "trim@0.0.1"
[3/5] Fetching packages...
info fsevents@2.1.2: The platform "linux" is incompatible with this module.
info "fsevents@2.1.2" is an optional dependency and failed compatibility check. Excluding it from installation.
[4/5] Linking dependencies...
warning " > @elastic/eui@30.6.0" has incorrect peer dependency "typescript@^3.7.2".
warning " > @elastic/apm-rum-react@1.2.5" has incorrect peer dependency "react-router-dom@^4.0.0".
warning "@storybook/addon-essentials > @storybook/addon-docs > @mdx-js/react@1.6.16" has incorrect peer dependency "react@^16.13.1".
warning " > react-shortcuts@2.0.1" has incorrect peer dependency "react@^0.14.8 || ^15".
warning " > react-shortcuts@2.0.1" has incorrect peer dependency "react-dom@^0.14.8 || ^15".
[5/5] Building fresh packages...

succ yarn.lock analysis completed without any issues
succ 18 bootstrap builds are cached
info [@kbn/analytics] running [kbn:bootstrap] script
succ [@kbn/analytics] bootstrap complete
info [@kbn/monaco] running [kbn:bootstrap] script
succ [@kbn/monaco] bootstrap complete
info [@kbn/ui-shared-deps] running [kbn:bootstrap] script
succ [@kbn/ui-shared-deps] bootstrap complete
info [@kbn/optimizer] running [kbn:bootstrap] script
succ [@kbn/optimizer] bootstrap complete
info [@kbn/plugin-helpers] running [kbn:bootstrap] script
info [@kbn/test] running [kbn:bootstrap] script
succ [@kbn/plugin-helpers] bootstrap complete
succ [@kbn/test] bootstrap complete
info [@kbn/es-archiver] running [kbn:bootstrap] script
info [x-pack] running [kbn:bootstrap] script
succ [x-pack] bootstrap complete
succ [@kbn/es-archiver] bootstrap complete
info [kibana] running [kbn:bootstrap] script
succ [kibana] bootstrap complete
Done in 432.11s.

Seems like I had to wait a long long time for Kibana to connect to Elasticsearch:

Here is the output:

[ec2-user@ip-172-31-24-126 kibana] yarn start --run-examples yarn run v1.22.10 node scripts/kibana --dev --run-examples
[2020-12-16T22:03:53.970Z][INFO ][plugins-service] Plugin initialization disabled.
[2020-12-16T22:03:54.033Z][WARN ][savedobjects-service] Skipping Saved Object migrations on startup. Note: Individual docu ments will still be migrated when read or written.
no-base-path ====================================================================================================
no-base-path Running Kibana in dev mode with --no-base-path disables several useful features and is not recommended
no-base-path ====================================================================================================
np bld log [22:03:58.180] [info][@kbn/optimizer] initialized, 74 bundles cached
np bld log [22:03:58.190] [warning][@kbn/optimizer] only building [v7dark,v7light] themes, customize with the KBN_OPT IMIZER_THEMES environment variable
np bld log [22:04:03.433] [info][@kbn/optimizer] starting worker [23 bundles]
np bld log [22:04:03.555] [info][@kbn/optimizer] starting worker [23 bundles]
watching for changes (8015 files)
server log [22:04:20.980] [info][plugins-service] Plugin "visTypeXy" is disabled.
server log [22:04:21.015] [info][plugins-service] Plugin "reporting" is disabled.
server log [22:04:22.384] [info][plugins-system] Setting up [106] plugins: [usageCollection,xpackLegacy,telemetryCol lectionManager,telemetry,telemetryCollectionXpack,kibanaUsageCollection,taskManager,licensing,globalSearch,globalSearchPro viders,code,securityOss,newsfeed,mapsLegacy,kibanaLegacy,routingExample,translations,share,legacyExport,embeddable,uiActio nsEnhanced,expressions,charts,esUiShared,bfetch,data,home,observability,console,consoleExtensions,apmOss,searchprofiler,pa inlessLab,grokdebugger,management,indexPatternManagement,advancedSettings,fileUpload,savedObjects,visualizations,visTypeMa rkdown,visTypeVislib,visTypeVega,visTypeTimeseries,visTypeTimeseriesEnhanced,visTypeTimelion,features,licenseManagement,da taEnhanced,watcher,canvas,visTypeTagcloud,visTypeTable,visTypeMetric,tileMap,regionMap,mapsOss,lensOss,inputControlVis,gra ph,timelion,dashboard,embeddableExamples,dashboardEnhanced,visualize,stateContainersExamples,searchExamples,discover,disco verEnhanced,savedObjectsManagement,spaces,security,savedObjectsTagging,maps,lens,lists,encryptedSavedObjects,dashboardMode ,cloud,upgradeAssistant,snapshotRestore,fleet,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycle Management,enterpriseSearch,ml,beatsManagement,transform,ingestPipelines,eventLog,actions,alerts,triggersActionsUi,stackAl erts,securitySolution,case,infra,monitoring,logstash,apm,alertingExample,uptime,bfetchExplorer]
server log [22:04:22.408] [info][plugins][taskManager] TaskManager is identified by the Kibana UUID: 64f54d40-ee3e-4 99b-9e95-f5dfe88ec3d8
server log [22:04:23.442] [warning][config][plugins][security] Session cookies will be transmitted over insecure con nections. This is not recommended.
server log [22:04:24.974] [info][monitoring][monitoring][plugins] config sourced from: production cluster
server log [22:04:26.438] [info][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kib ana before starting saved objects migrations...
server log [22:04:26.917] [info][savedobjects-service] Starting saved objects migrations
server log [22:04:27.351] [info][savedobjects-service] Creating index .kibana_1.
server log [22:04:27.461] [info][savedobjects-service] Creating index .kibana_task_manager_1.
server log [22:04:28.304] [info][savedobjects-service] Pointing alias .kibana to .kibana_1.
server log [22:04:28.635] [info][savedobjects-service] Pointing alias .kibana_task_manager to .kibana_task_manager_1 .
server log [22:04:28.934] [info][savedobjects-service] Finished in 1604ms.
server log [22:04:29.019] [info][savedobjects-service] Finished in 1558ms.
server log [22:04:29.381] [info][plugins-system] Starting [106] plugins: [usageCollection,xpackLegacy,telemetryColle ctionManager,telemetry,telemetryCollectionXpack,kibanaUsageCollection,taskManager,licensing,globalSearch,globalSearchProvi ders,code,securityOss,newsfeed,mapsLegacy,kibanaLegacy,routingExample,translations,share,legacyExport,embeddable,uiActions Enhanced,expressions,charts,esUiShared,bfetch,data,home,observability,console,consoleExtensions,apmOss,searchprofiler,pain lessLab,grokdebugger,management,indexPatternManagement,advancedSettings,fileUpload,savedObjects,visualizations,visTypeMark down,visTypeVislib,visTypeVega,visTypeTimeseries,visTypeTimeseriesEnhanced,visTypeTimelion,features,licenseManagement,data Enhanced,watcher,canvas,visTypeTagcloud,visTypeTable,visTypeMetric,tileMap,regionMap,mapsOss,lensOss,inputControlVis,graph ,timelion,dashboard,embeddableExamples,dashboardEnhanced,visualize,stateContainersExamples,searchExamples,discover,discove rEnhanced,savedObjectsManagement,spaces,security,savedObjectsTagging,maps,lens,lists,encryptedSavedObjects,dashboardMode,c loud,upgradeAssistant,snapshotRestore,fleet,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleMa nagement,enterpriseSearch,ml,beatsManagement,transform,ingestPipelines,eventLog,actions,alerts,triggersActionsUi,stackAler ts,securitySolution,case,infra,monitoring,logstash,apm,alertingExample,uptime,bfetchExplorer]
[BABEL] Note: The code generator has deoptimised the styling of /home/ec2-user/repos/kibana/x-pack/plugins/security_solut ion/server/utils/beat_schema/fields.ts as it exceeds the max of 500KB.
server log [22:05:03.030] [info][listening] Server running at http://0.0.0.0:5601
server log [22:05:10.506] [info][server][Kibana][http] http server running at http://0.0.0.0:5601
server log [22:05:10.599] [warning][licensing][plugins] License information could not be obtained from Elasticsearch due to Error: Request Timeout after 30000ms error
server log [22:05:10.615] [info][plugins][searchprofiler] You cannot use searchprofiler because license information is not available at this time.
server log [22:05:10.622] [info][painlessLab][plugins] You cannot use painlessLab because license information is not available at this time.
server log [22:05:10.634] [info][plugins][watcher] You cannot use watcher because license information is not availab le at this time.
server log [22:05:10.640] [info][plugins][snapshotRestore] You cannot use snapshot_restore because license informati on is not available at this time.
server log [22:05:10.640] [info][indexManagement][plugins] You cannot use index_management because license informati on is not available at this time.
server log [22:05:10.640] [info][plugins][rollup] You cannot use rollup because license information is not available at this time.
server log [22:05:10.640] [info][plugins][remoteClusters] You cannot use Remote Clusters because license information is not available at this time.
server log [22:05:10.641] [info][crossClusterReplication][plugins] You cannot use crossClusterReplication because li cense information is not available at this time.
server log [22:05:10.641] [info][indexLifecycleManagement][plugins] You cannot use index_lifecycle_management becaus e license information is not available at this time.
server log [22:05:10.654] [info][plugins][transform] You cannot use transform because license information is not ava ilable at this time.
server log [22:05:10.654] [info][ingestPipelines][plugins] You cannot use ingest_pipelines because license informati on is not available at this time.
server log [22:05:10.655] [info][kibana-monitoring][monitoring][monitoring][plugins] Monitoring status upload endpoi nt is not enabled in Elasticsearch:Monitoring stats collection is stopped
Unhandled Promise rejection detected:

TimeoutError: Request timed out
at ClientRequest.onTimeout (/home/ec2-user/repos/kibana/node_modules/@elastic/elasticsearch/lib/Connection.js:108:16)
at ClientRequest.emit (events.js:315:20)
at Socket.emitRequestTimeout (_http_client.js:784:9)
at Object.onceWrapper (events.js:421:28)
at Socket.emit (events.js:327:22)
at Socket._onTimeout (net.js:483:8)
at listOnTimeout (internal/timers.js:554:17)
at processTimers (internal/timers.js:497:7) {
meta: {
body: null,
statusCode: null,
headers: null,
meta: {
context: null,
request: [Object],
name: 'elasticsearch-js',
connection: [Object],
attempts: 0,
aborted: false
}
},
isBoom: true,
isServer: true,
data: null,
output: {
statusCode: 503,
payload: {
statusCode: 503,
error: 'Service Unavailable',
message: 'Request timed out'
},
headers: {}
},
[Symbol(SavedObjectsClientErrorCode)]: 'SavedObjectsClient/esUnavailable'
}

Terminating process...
server crashed with status code 1

I've noticed a warning in elasticsearch logs, but it starts.

[ec2-user@ip-172-31-24-126 kibana]$ curl -u elastic:changeme localhost:9200
{
"name" : "ip-172-31-24-126.us-west-1.compute.internal",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "yIw4NWtgSGCRF6_2SPjIjA",
"version" : {
"number" : "8.0.0-SNAPSHOT",
"build_flavor" : "default",
"build_type" : "tar",
"build_hash" : "7011cbac5d9a2f02deb090d057d05a97a0e9ad89",
"build_date" : "2020-12-14T19:32:07.674355Z",
"build_snapshot" : true,
"lucene_version" : "8.7.0",
"minimum_wire_compatibility_version" : "7.11.0",
"minimum_index_compatibility_version" : "7.0.0"
},
"tagline" : "You Know, for Search"
}

I could correct this issue it by adding discovery.type: single-node to the elasticsearch.yml file, but the file is temporary and will be overridden when I start elasticsearch => yarn es snapshot --license trial.

│ info [o.e.b.BootstrapChecks] [ip-172-31-24-126.us-west-1.compute.internal] the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
│ info [o.e.c.c.ClusterBootstrapService] [ip-172-31-24-126.us-west-1.compute.internal] no discovery configuration found, will perform best-effort cluster bootstrapping after [3s] unless existing master is discovered

I've try restarting everything again and now I receive a response back when I goto instance_ip:5601, but I still get an error:
[ec2-user@ip-172-31-24-126 kibana] yarn start --run-examples yarn run v1.22.10 node scripts/kibana --dev --run-examples
[2020-12-16T22:19:27.085Z][INFO ][plugins-service] Plugin initialization disabled.
[2020-12-16T22:19:27.148Z][WARN ][savedobjects-service] Skipping Saved Object migrations on startup. Note: Individual documents w ill still be migrated when read or written.
no-base-path ====================================================================================================
no-base-path Running Kibana in dev mode with --no-base-path disables several useful features and is not recommended
no-base-path ====================================================================================================
np bld log [22:19:30.828] [info][@kbn/optimizer] initialized, 74 bundles cached
np bld log [22:19:30.836] [warning][@kbn/optimizer] only building [v7dark,v7light] themes, customize with the KBN_OPTIMIZER_ THEMES environment variable
np bld log [22:19:36.343] [info][@kbn/optimizer] starting worker [23 bundles]
np bld log [22:19:36.344] [info][@kbn/optimizer] starting worker [23 bundles]
watching for changes (8015 files)
server log [22:19:54.361] [info][plugins-service] Plugin "visTypeXy" is disabled.
server log [22:19:54.404] [info][plugins-service] Plugin "reporting" is disabled.
server log [22:19:55.991] [info][plugins-system] Setting up [106] plugins: [usageCollection,xpackLegacy,telemetryCollection Manager,telemetry,telemetryCollectionXpack,kibanaUsageCollection,taskManager,licensing,globalSearch,globalSearchProviders,code,se curityOss,newsfeed,mapsLegacy,kibanaLegacy,routingExample,translations,share,legacyExport,embeddable,uiActionsEnhanced,expression s,charts,esUiShared,bfetch,data,home,observability,console,consoleExtensions,apmOss,searchprofiler,painlessLab,grokdebugger,manag ement,indexPatternManagement,advancedSettings,fileUpload,savedObjects,visualizations,visTypeVislib,visTypeVega,visTypeTimeseries, visTypeTimeseriesEnhanced,visTypeTimelion,features,licenseManagement,dataEnhanced,watcher,canvas,visTypeTagcloud,visTypeTable,vis TypeMarkdown,visTypeMetric,tileMap,regionMap,mapsOss,lensOss,inputControlVis,graph,timelion,dashboard,embeddableExamples,dashboar dEnhanced,visualize,stateContainersExamples,searchExamples,discover,discoverEnhanced,savedObjectsManagement,spaces,security,saved ObjectsTagging,maps,lens,lists,encryptedSavedObjects,dashboardMode,cloud,upgradeAssistant,snapshotRestore,fleet,indexManagement,r ollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,enterpriseSearch,ml,beatsManagement,transform,ingestPipelin es,eventLog,actions,alerts,triggersActionsUi,stackAlerts,securitySolution,case,infra,monitoring,logstash,apm,alertingExample,upti me,bfetchExplorer]
server log [22:19:56.048] [info][plugins][taskManager] TaskManager is identified by the Kibana UUID: 64f54d40-ee3e-499b-9e9 5-f5dfe88ec3d8
server log [22:19:57.558] [warning][config][plugins][security] Session cookies will be transmitted over insecure connection s. This is not recommended.
server log [22:19:59.509] [info][monitoring][monitoring][plugins] config sourced from: production cluster
server log [22:20:01.899] [info][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana bef ore starting saved objects migrations...
server log [22:20:02.588] [info][savedobjects-service] Starting saved objects migrations
server log [22:20:03.036] [info][savedobjects-service] Creating index .kibana_1.
server log [22:20:03.053] [info][savedobjects-service] Creating index .kibana_task_manager_1.
server log [22:20:04.121] [info][savedobjects-service] Pointing alias .kibana to .kibana_1.
server log [22:20:04.495] [info][savedobjects-service] Pointing alias .kibana_task_manager to .kibana_task_manager_1.
server log [22:20:04.803] [info][savedobjects-service] Finished in 1797ms.
server log [22:20:04.855] [info][savedobjects-service] Finished in 1802ms.
server log [22:20:05.271] [info][plugins-system] Starting [106] plugins: [usageCollection,xpackLegacy,telemetryCollectionMa nager,telemetry,telemetryCollectionXpack,kibanaUsageCollection,taskManager,licensing,globalSearch,globalSearchProviders,code,secu rityOss,newsfeed,mapsLegacy,kibanaLegacy,routingExample,translations,share,legacyExport,embeddable,uiActionsEnhanced,expressions, charts,esUiShared,bfetch,data,home,observability,console,consoleExtensions,apmOss,searchprofiler,painlessLab,grokdebugger,managem ent,indexPatternManagement,advancedSettings,fileUpload,savedObjects,visualizations,visTypeVislib,visTypeVega,visTypeTimeseries,vi sTypeTimeseriesEnhanced,visTypeTimelion,features,licenseManagement,dataEnhanced,watcher,canvas,visTypeTagcloud,visTypeTable,visTy peMarkdown,visTypeMetric,tileMap,regionMap,mapsOss,lensOss,inputControlVis,graph,timelion,dashboard,embeddableExamples,dashboardE nhanced,visualize,stateContainersExamples,searchExamples,discover,discoverEnhanced,savedObjectsManagement,spaces,security,savedOb jectsTagging,maps,lens,lists,encryptedSavedObjects,dashboardMode,cloud,upgradeAssistant,snapshotRestore,fleet,indexManagement,rol lup,remoteClusters,crossClusterReplication,indexLifecycleManagement,enterpriseSearch,ml,beatsManagement,transform,ingestPipelines ,eventLog,actions,alerts,triggersActionsUi,stackAlerts,securitySolution,case,infra,monitoring,logstash,apm,alertingExample,uptime ,bfetchExplorer]
server log [22:20:09.105] [info][plugins][watcher] Enabling Watcher plugin.
server log [22:20:09.128] [info][kibana-monitoring][monitoring][monitoring][plugins] Starting monitoring stats collection
[BABEL] Note: The code generator has deoptimised the styling of /home/ec2-user/repos/kibana/x-pack/plugins/security_solution/ser ver/utils/beat_schema/fields.ts as it exceeds the max of 500KB.
server log [22:20:39.056] [info][listening] Server running at http://0.0.0.0:5601
server log [22:20:45.945] [info][server][Kibana][http] http server running at http://0.0.0.0:5601
[BABEL] Note: The code generator has deoptimised the styling of /home/ec2-user/repos/kibana/x-pack/plugins/canvas/server/templat es/pitch_presentation.ts as it exceeds the max of 500KB.
server log [22:20:50.537] [error][ml][plugins] Error Initializing jobs {"name":"TimeoutError","meta":{"body":null,"statusCo de":null,"headers":null,"meta":{"context":null,"request":{"params":{"method":"POST","path":"/.kibana/_search","body":"{"seq_no_p rimary_term":true,"query":{"bool":{"filter":[{"bool":{"should":[{"bool":{"must":[{"term":{"type":"ml-job"}}] ,"must_not":[{"exists":{"field":"namespace"}}]}}],"minimum_should_match":1}}]}}}","querystring":"size=0&from=0&rest_tot al_hits_as_int=true","headers":{"user-agent":"elasticsearch-js/7.10.0 (linux 4.14.209-160.335.amzn2.x86_64-x64; Node.js v14.15.2) ","x-elastic-product-origin":"kibana","content-type":"application/json","content-length":"198"},"timeout":30000},"options":{"maxR etries":0,"ignore":[404]},"id":21},"name":"elasticsearch-js","connection":{"url":"http://localhost:9200/","id":"http://localhost: 9200/","headers":{},"deadCount":2,"resurrectTimeout":1608157370536,"_openRequests":18,"status":"dead","roles":{"master":true,"dat a":true,"ingest":true,"ml":false}},"attempts":0,"aborted":false}},"isBoom":true,"isServer":true,"data":null,"output":{"statusCode ":503,"payload":{"statusCode":503,"error":"Service Unavailable","message":"Request timed out"},"headers":{}}}
server log [22:20:50.549] [error][index][plugins][security][session] Failed to check if session index template exists: Requ est Timeout after 30000ms
server log [22:20:50.563] [warning][kibana-monitoring][monitoring][monitoring][plugins] Error: Request Timeout after 30000m s
at /home/ec2-user/repos/kibana/node_modules/elasticsearch/src/lib/transport.js:397:9
at Timeout. (/home/ec2-user/repos/kibana/node_modules/elasticsearch/src/lib/transport.js:429:7)
at listOnTimeout (internal/timers.js:554:17)
at processTimers (internal/timers.js:497:7)
server log [22:20:50.577] [warning][kibana-monitoring][monitoring][monitoring][plugins] Unable to bulk upload the stats pay load to the local cluster
server log [22:20:50.586] [error][plugins][security][session] Failed to retrieve session index cleanup task: Request timed out
server log [22:20:50.599] [error][plugins][securitySolution][telemetry_events] Error scheduling task, received Request time d out

What do you get with just a normal yarn start without the examples added?

The messages starting with "Creating index .kibana_1." seem to indicate that it can connect to elasticsearch.

But then the message "[error][ml][plugins] Error Initializing jobs" seems to be that it can no longer connect to elasticsearch. Could elasticsearch have gone down between the two messages?

Also, one thing I've been doing right after a yarn kbn bootstrap is node scripts/build_kibana_platform_plugins; this will do all the plugin builds, so that when you run yarn start it won't have to do the builds. Even with having yarn start do the builds, it's only the first time, but ... I just prefer to always do all the builds at the same time, so I never have a long wait for Kibana to come up.

@arron-nimocks
With yarn start I can reach Kibana:

Think that just means at least one of the plugins in /examples/ doesn't work with the current version.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.