Moving Kibana 7.13.3 development environment

Elastic Stack Kibana
1

Hello all,

I hope you are doing well! I have a question about my kibana development environment. I have developed kibana from source on one workstation and copied it from there and pasted it on another work station. it isn't working at another place and I can not develop it there because of security restrictions. Is there any other way? should it work there?

Thank you in advance!

2

Hey -

Can you share a little more info? What security restrictions are in place? What error logs are you seeing?

I'm trying to think what we'd see, but without logs it's hard to debug. Network restrictions would cause issues. We'd probably see fs errors under selinux.

3

Hello @jbudz ,

My appologies, these are just proxy setups, nothing else. but after moving whole development environment when I run yarn es snapshot, it simply gives the following error. This is not supposed to happen because the same setup works on another workstation.

$ yarn es snapshot
yarn run v1.22.5
$ node scripts/es snapshot
Error: Cannot find module '@kbn/es'
Require stack:
- C:\TEMP\dev-kibana7133\kibana\scripts\es.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:902:15)
    at Module.Hook._require.Module.require (C:\TEMP\dev-kibana7133\kibana\node_m
odules\require-in-the-middle\index.js:61:29)
    at require (internal/modules/cjs/helpers.js:92:18)
    at Object.<anonymous> (C:\TEMP\dev-kibana7133\kibana\scripts\es.js:13:13)
    at Module._compile (internal/modules/cjs/loader.js:1085:14)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
    at Module.load (internal/modules/cjs/loader.js:950:32)
    at Function.Module._load (internal/modules/cjs/loader.js:790:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js
:76:12)
    at internal/main/run_main_module.js:17:47
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this comm
and.

Thanks.

4

Can you try running yarn kbn bootstrap and share the output?

5

Hello @jbudz,

Here it is.

$ yarn kbn bootstrap
yarn run v1.22.5
$ node scripts/kbn bootstrap
 info [bazel_tools] installing Bazel tools
 warn failed to reach ci-stats service [reason=no response], retrying in 1 seconds
 warn failed to reach ci-stats service [reason=no response], retrying in 2 seconds
 warn failed to reach ci-stats service [reason=no response], retrying in 3 seconds
 warn failed to reach ci-stats service [reason=no response], retrying in 4 seconds
 warn unable to report 1 timing, failed to reach ci-stats service too many times
ERROR [bootstrap] failed:
ERROR Error: Command failed with exit code 1: npm install --global @bazel/bazelisk@1.7.5
      npm ERR! code ENOTFOUND
      npm ERR! errno ENOTFOUND
      npm ERR! network request to https://registry.yarnpkg.com/@bazel%2fbazelisk failed, reason: getaddrinfo ENOTFOUND registry.yarnpkg.com
      npm ERR! network This is a problem related to network connectivity.
      npm ERR! network In most cases you are behind a proxy or have bad network settings.
      npm ERR! network
      npm ERR! network If you are behind a proxy, please make sure that the
      npm ERR! network 'proxy' config is set properly.  See: 'npm help config'

      npm ERR! A complete log of this run can be found in:
      npm ERR!     C:\Users\MMS5822\AppData\Roaming\npm-cache\_logs\2021-07-12T17_23_30_528Z-debug.log
          at makeError (C:\TEMP\dev-kibana7133\kibana\packages\kbn-pm\dist\index.js:35184:11)
          at handlePromise (C:\TEMP\dev-kibana7133\kibana\packages\kbn-pm\dist\index.js:34095:26)
          at processTicksAndRejections (internal/process/task_queues.js:95:5)
          at async installBazelTools (C:\TEMP\dev-kibana7133\kibana\packages\kbn-pm\dist\index.js:48420:5)
          at async Object.run (C:\TEMP\dev-kibana7133\kibana\packages\kbn-pm\dist\index.js:8937:5)
          at async runCommand (C:\TEMP\dev-kibana7133\kibana\packages\kbn-pm\dist\index.js:59803:5)
          at async Module.run (C:\TEMP\dev-kibana7133\kibana\packages\kbn-pm\dist\index.js:242:3)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Thanks

6

I think I know the problem, why I could not run yarn es snapshot.

when I copied the development environment, it copied all the directories form
kibana > node_modules directory but the data inside those directories.

is that supposed to happen? I am trying to copy everything again and see what happens. Thanks

7

Hello @jbudz,

I tried copying the development environment again but it does not copy contents of node_modules. Is that supposed to happen? It does not even give me any errors. Please help!

I can not remove proxy on my other workstation where my plugin is. So I can not run yarn kbn bootstrap there.

as per this article, development is not possible when behind corporate proxy.

Thanks

8

How are you copying the files? Yes, if we want to try and run a copy without bootstrapping we'll need everything.

It's going to be difficult to run a development version of kibana without access to external services otherwise.

We have a number of downloads - elasticsearch, chromium, node_modules, bazel caches, typescript reference caches, and so on.

9

Hello @jbudz,

So, as I said I am not able to create kibana development envrionment on my workstation because of corporate proxy.

on other workstation, I downloaded Kibana 7.13.3 from source from github and ran yarn kbn bootstrap there and created a custom plugin. after that I was able to run elasticsearch and kibana.

So I can created the whole kibana folder into USB drive then I found out it copies all the folders inside @kbn directory but not the contents. I hope this makes sense.

in addition to that, I copied everything manually in USB drive and running it again on my workstation. I am getting following errors.

$ yarn es snapshot
yarn run v1.22.5
$ node scripts/es snapshot
 info Installing from snapshot
   │ info version: 7.13.3
   │ info install path: C:\TEMP\dev-kibana7133\kibana\.es\7.13.3
   │ info license: basic
   │ info Downloading snapshot manifest from https://storage.googleapis.com/kibana-ci-es-snapshots-daily/7.13.3/manifest-latest-verified.json
   │ warn ...failure, retrying in 5 seconds: request to https://storage.googleapis.com/kibana-ci-es-snapshots-daily/7.13.3/manifest-latest-verified.json failed, reason: get
addrinfo ENOTFOUND storage.googleapis.com
   │ info ...retrying
   │ warn ...failure, retrying in 5 seconds: request to https://storage.googleapis.com/kibana-ci-es-snapshots-daily/7.13.3/manifest-latest-verified.json failed, reason: get
addrinfo ENOTFOUND storage.googleapis.com
   │ info ...retrying
   │ warn ...failure, retrying in 5 seconds: request to https://storage.googleapis.com/kibana-ci-es-snapshots-daily/7.13.3/manifest-latest-verified.json failed, reason: get
addrinfo ENOTFOUND storage.googleapis.com
   │ info ...retrying
   │ warn ...failure, retrying in 5 seconds: request to https://storage.googleapis.com/kibana-ci-es-snapshots-daily/7.13.3/manifest-latest-verified.json failed, reason: get
addrinfo ENOTFOUND storage.googleapis.com
   │ info ...retrying
   │ERROR Unhandled error
   │ERROR FetchError: request to https://storage.googleapis.com/kibana-ci-es-snapshots-daily/7.13.3/manifest-latest-verified.json failed, reason: getaddrinfo ENOTFOUND stor
age.googleapis.com
   │          at ClientRequest.<anonymous> (C:\TEMP\dev-kibana7133\kibana\node_modules\node-fetch\lib\index.js:1461:11)
   │          at ClientRequest.emit (events.js:375:28)
   │          at TLSSocket.socketErrorListener (_http_client.js:475:9)
   │          at TLSSocket.emit (events.js:375:28)
   │          at emitErrorNT (internal/streams/destroy.js:106:8)
   │          at emitErrorCloseNT (internal/streams/destroy.js:74:3)
   │          at processTicksAndRejections (internal/process/task_queues.js:82:21)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Thank again Jon for helping me.

10

Hi Akhil,

Have you tried setting the proxy for yarn and npm?

yarn config set proxy http://username:password@host:port
yarn config set https-proxy http://username:password@host:port

In case you use a self-signed CA in your environment, you may have to add the CA in your trust store for npm or disable strict ssl. This would be the best option to bootstrap the env on your machine.

npm config set proxy http://username:password@host:port
npm config set https-proxy http://username:password@host:port
npm set strict-ssl false
11

Hello @NerdSec,

I tried setting up the proxy as well but nothing seems to work.

Thanks

12

Hi @NerdSec,

I think the proxy settings are working now. but now I am getting these errors.

$ yarn kbn bootstrap
yarn run v1.22.5
$ node scripts/kbn bootstrap
 info [bazel_tools] installing Bazel tools
 succ [bazel_tools] all bazel tools are correctly installed
 info [bazel] FATAL: AcquireLock(c:\temp\kibana-development\kibana\unc\sh12cffp0001\users-d\ms582\_bazel_ms582\bvg2zzma\lock): CreateFile failed: (error: 3): The system c
annot find the path specified.
 info [bazel]
 warn failed to reach ci-stats service [reason=no response], retrying in 1 seconds
 warn failed to reach ci-stats service [reason=no response], retrying in 2 seconds
 warn failed to reach ci-stats service [reason=no response], retrying in 3 seconds
 warn failed to reach ci-stats service [reason=no response], retrying in 4 seconds
 warn unable to report 1 timing, failed to reach ci-stats service too many times
ERROR [bootstrap] failed:
ERROR The bazel command that was running failed to complete.
error Command failed with exit code 1.

Thanks for your help!

13

Did you do a clean install or on a copied directory with your plugin files? A general approach should be you copy your plugin code and setup a new dev env using the kibana plugin guide.

14

Hi Nachiket,

right now I am not copying the plugin code. As I was able to bypass proxy, I downloaded kibana 7.13.3 from the source and trying to setup a dev env again.
while doing that, I am getting the error above.

Thanks

15

Hi Akhil,

What shell are you using? Could it be related to this?
https://docs.bazel.build/versions/main/windows.html#running-bazel-msys2-shell-vs-command-prompt-vs-powershell

Regards,
Nachiket

16

Hi Nachiket,

I am using gitbash. As per this article, it says that it is recommended using cmd or powershell for bazel but kibana development is recommended using gitbash.

but you are right because in powershell and cmd the errors are somewhat different.

PS C:\TEMP\kibana-development\kibana> yarn kbn bootstrap
yarn run v1.21.1
$ node scripts/kbn bootstrap
 info [bazel_tools] installing Bazel tools
 succ [bazel_tools] all bazel tools are correctly installed
 info [bazel] INFO: Invocation ID: 40268a01-6f12-41ce-987e-8b0c1c077899
 info [bazel] INFO: Repository build_bazel_rules_nodejs instantiated at:
 info [bazel]   C:/temp/kibana-development/kibana/WORKSPACE.bazel:11:13: in <toplevel>
 info [bazel] Repository rule http_archive defined at:
 info [bazel]   C:/users/ms82/_bazel_ms82/bvg2zzma/external/bazel_tools/tools/build_defs/repo/http.bzl:336:31: in <toplevel>
 info [bazel] WARNING: Download from https://github.com/bazelbuild/rules_nodejs/releases/download/3.2.3/rules_nodejs-3.2.3.tar.gz failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException Unknown host: github.com
 info [bazel] ERROR: An error occurred during the fetch of repository 'build_bazel_rules_nodejs':
 info [bazel]    Traceback (most recent call last):
 info [bazel]   File "C:/users/ms82/_bazel_ms82/bvg2zzma/external/bazel_tools/tools/build_defs/repo/http.bzl", line 111, column 45, in _http_archive_impl
 info [bazel]           download_info = ctx.download_and_extract(
 info [bazel] Error in download_and_extract: java.io.IOException: Error downloading [https://github.com/bazelbuild/rules_nodejs/releases/download/3.2.3/rules_nodejs-3.2.3.tar.gz] to C:/users/ms82/_bazel_ms82/bvg2zzma/external/build_bazel_rules_nodejs/temp5743307366720225877/rules_nodejs-3.2.3.tar.gz: Unknown host: github.com
 info [bazel] ERROR: Error fetching repository: Traceback (most recent call last):
 info [bazel]   File "C:/users/ms82/_bazel_ms82/bvg2zzma/external/bazel_tools/tools/build_defs/repo/http.bzl", line 111, column 45, in _http_archive_impl
 info [bazel]           download_info = ctx.download_and_extract(
 info [bazel] Error in download_and_extract: java.io.IOException: Error downloading [https://github.com/bazelbuild/rules_nodejs/releases/download/3.2.3/rules_nodejs-3.2.3.tar.gz] to C:/users/ms82/_bazel_ms52/bvg2zzma/external/build_bazel_rules_nodejs/temp5743307366720225877/rules_nodejs-3.2.3.tar.gz: Unknown host: github.com
 info [bazel] ERROR: no such package '@build_bazel_rules_nodejs//': java.io.IOException: Error downloading [https://github.com/bazelbuild/rules_nodejs/releases/download/3.2.3/rules_nodejs-3.2.3.tar.gz] to C:/users/ms82/_bazel_ms82/bvg2zzma/external/build_bazel_rules_nodejs/temp5743307366720225877/rules_nodejs-3.2.3.tar.gz: Unknown host: github.com
 info [bazel] INFO: Elapsed time: 1.475s
 info [bazel] INFO: 0 processes.
 info [bazel]
 warn failed to reach ci-stats service [reason=no response], retrying in 1 seconds
 warn failed to reach ci-stats service [reason=no response], retrying in 2 seconds
 warn failed to reach ci-stats service [reason=no response], retrying in 3 seconds
 warn failed to reach ci-stats service [reason=no response], retrying in 4 seconds
 warn unable to report 1 timing, failed to reach ci-stats service too many times
ERROR [bootstrap] failed:
ERROR The bazel command that was running failed to complete.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Thanks

17

Do you have access to github? Is the DNS resolution to github.com happening on your machine?

18

Hi Nachiket,

yes, I do have access to GitHub as I cloned the repository through gitbash from GitHub.

Thanks,
Akhil

19

Hi @NerdSec,

Thanks for helping me man! it worked. I deleted proxy settings, added them again.
copied developed environment again properly. I am still getting the following error but it works.

 warn failed to reach ci-stats service [reason=no response], retrying in 1 seconds

Thanks.

20

I believe you can ignore that... It is just the stat collection service of Kibana. Not sure, maybe someone from Kibana team can confirm!