Open Source Curation

Elastic Community and Ecosystem
1

I'm responsible for curating open source for a project that uses Elasticsearch and Kibana. My specific group requires us to start with source code for the product and ALL dependencies and build the final binaries OFFLINE.

I may have figured out how to build Elasticsearch. Download and archive source for Elasticsearch and Gradle, let Gradle download hundreds of Java modules (jar/pom) and archive them, then place it all into a single container and build away. I actually get installable packages in the end. Only have to process licenses for all the Java modules.

Kibana is easier in some regards but more difficult overall and likely the deal breaker.

  1. Downloading Kibana source tarballs does not work in the build process. You need to clone the git repository since the build process runs some git commands and fails in an unclear manner. You can't use the tarball with a 'git init' method either. Must be a clone from what I can tell.

  2. The CONTRIBUTING.md file steers users astray with unnecessary steps for starting Elasticsearch which discuss.elastic.co discussions clarify can be skipped, or populating Elasticsearch with data, or using a script for nvm to download node.js/npm. If the instructions said simple to a) download version x.y.z of node.js, b) cd to Kibana, c) run 'npm install', d) run 'npm update', e) ignore specific errors otherwise panic, then f) 'npm run build xxxxx' to get installable packages, I would have saved a few days. Bad instructions waste time and frustrate potential users/customers.

  3. NPM installed cleanly, but the 'npm install' didn't install all dependencies without 'UNMET depencencies'. Even 'npm update' didn't resolve all the issues. The issue with grunt-angular-translate requiring grunt version 0.4.0 remains and the only discuss.elastic.co discussion suggested --force or ignoring the issue because some specific build target didn't hit the dependency. I wasted a lot of time here too thinking this grunt-angular-translate issue was the cause of all other issues. For curating open source software, npm is easy to work with for packages/versions/licenses.

  1. The instructions weren't clear what it takes to make an installable package. I guessed that it was 'npm run build -- --skip-os-packages'. During the build of that target with INTERNET connectivity, it appears to complete regardless of the error:

Loading "ui_framework.js" tasks...ERROR
Error: ENOENT: no such file or directory, scandir '/opt/kibana/git/kibana/node_modules/node-sass/vendor'

But it downloads more stuff! :frowning:

Running "_build:downloadNodeBuilds" task
Downloading darwin-x86_64 and corresponding sha
Downloading linux-x86 and corresponding sha
Downloading windows-x86 and corresponding sha
Downloading linux-x86_64 and corresponding sha
darwin-x86_64 downloaded and verified
linux-x86 downloaded and verified
linux-x86_64 downloaded and verified
windows-x86 downloaded and verified

Where are my rpm/deb packages???

I've done a 'docker disable network bridge build_kibana' to keep the build from going to the Internet, but it fails. To me, this suggests NO support for building offline.

Running "_build:downloadNodeBuilds" task
Warning: Client request error: getaddrinfo EAI_AGAIN nodejs.org:443 Use --force to continue.

Using --force to see what else breaks, sure enough there is more Internet activity as these failures show:

Running "_build:downloadNodeBuilds" task
Warning: Client request error: getaddrinfo EAI_AGAIN nodejs.org:443 Used --force, continuing.

And:

Running "_build:installNpmDeps" task
Warning: Command failed: npm install --production --no-optional
npm ERR! git fetch -a origin (git://github.com/elastic/webpack.git) fatal: unable to connect to github.com:
npm ERR! git fetch -a origin (git://github.com/elastic/webpack.git) github.com: Temporary failure in name resolution
npm ERR! git clone --template=/root/.npm/_git-remotes/_templates --mirror git@github.com:elastic/webpack.git /root/.npm/_git-remotes/git-github-com-elastic-webpack-git-fix-query-params-for-aliased-loaders-696abd51: Cloning into bare repository '/root/.npm/_git-remotes/git-github-com-elastic-webpack-git-fix-query-params-for-aliased-loaders-696abd51'...
npm ERR! git clone --template=/root/.npm/_git-remotes/_templates --mirror git@github.com:elastic/webpack.git /root/.npm/_git-remotes/git-github-com-elastic-webpack-git-fix-query-params-for-aliased-loaders-696abd51: error: cannot run ssh: No such file or directory
npm ERR! git clone --template=/root/.npm/_git-remotes/_templates --mirror git@github.com:elastic/webpack.git /root/.npm/_git-remotes/git-github-com-elastic-webpack-git-fix-query-params-for-aliased-loaders-696abd51: fatal: unable to fork

In the end, it's still not clear how to make installable packages for Kibana. They are included in the official version of Kibana, so I assume there is a make option for them somewhere, but can't find it anywhere in the source distribution.

My take away is that by Elastic forcing the use of gradle/npm they move the responsibility of maintaining packages/versions/licenses to me, but with the added mess of a complicated build environment and lack of specific package/version manifests. We couldn't use the binary distribute before due to group policy, but now the binary distribution is out because the npm packages included in it are not identified clearly.

I don't mind doing a little work to benefit from a great product, but Elasticsearch and Kibana don't build or test cleanly due to the dynamic nature of these package eco-systems. I found little support for building these products after reading through CONTRIBUTING.md, included docs, web searches, Stack Exchange, and discuss.elastic.co.

Someone, please tell me I got it wrong...

2

I can only speak for the Kibana build process, but I can confirm that we have no mechanism in place to build Kibana without internet access. As you discovered, we do not ship copies of our dependencies in git, so those dependencies must be retrieved remotely.

I'm not really sure how it would work off-hand, but we'd welcome a feature request on github for the ability to create kibana builds using dependencies that are cached locally.

Downloading Kibana source tarballs does not work in the build process. You need to clone the git repository since the build process runs some git commands and fails in an unclear manner. You can't use the tarball with a 'git init' method either. Must be a clone from what I can tell.

This is correct. Git is a required part of both the development and build processes for Kibana.

The CONTRIBUTING.md file steers users astray with unnecessary steps for starting Elasticsearch which discuss.elastic.co discussions clarify can be skipped, or populating Elasticsearch with data, or using a script for nvm to download node.js/npm. If the instructions said simple to a) download version x.y.z of node.js, b) cd to Kibana, c) run 'npm install', d) run 'npm update', e) ignore specific errors otherwise panic, then f) 'npm run build xxxxx' to get installable packages, I would have saved a few days. Bad instructions waste time and frustrate potential users/customers.

The contributing guide is really focused on people getting Kibana set up for local development in order to contribute changes back upstream, which is why the instructions focus on tools like using nvm and getting a dev-only elasticsearch node running locally, but it sounds like you would have benefited from documentation focused specifically on building Kibana from source - does that sound accurate? This isn't a common ask, but it's still a pretty good idea regardless, since we don't really want anyone wasting their time going down a rabbit hole without hope of success. Our docs are in the repo, so we'd welcome a enhancement request on github for this with your advice based on the experience you had here.

NPM installed cleanly, but the 'npm install' didn't install all dependencies without 'UNMET depencencies'. Even 'npm update' didn't resolve all the issues. The issue with grunt-angular-translate requiring grunt version 0.4.0 remains and the only discuss.elastic.co discussion suggested --force or ignoring the issue because some specific build target didn't hit the dependency. I wasted a lot of time here too thinking this grunt-angular-translate issue was the cause of all other issues. For curating open source software, npm is easy to work with for packages/versions/licenses.

Sorry about the confusion that the angular-translate warnings caused. There's a PR up to remove this dependency entirely, which is probably why the warnings have just been ignored up to this point: https://github.com/elastic/kibana/pull/13066

The instructions weren't clear what it takes to make an installable package. I guessed that it was 'npm run build -- --skip-os-packages' ... Where are my rpm/deb packages???

The default for the build task is to build all of the baseline artifacts for Kibana, which includes deb and rpm packages. The --skip-os-packages specifically disables the building of rpm/deb packages, which is why none were created in your example. There are OS-specific dependencies that must be installed on the host machine to create the deb/rpm packages, and since this task runs directly on the host machine, we provide this flag so macos and windows users can still build the rest of the packages.

now the binary distribution is out because the npm packages included in it are not identified clearly

We include a NOTICE.txt file in the root of the repository that includes all of the license information from all dependencies in the project, including all of the npm dependencies. I hope this helps provide insight in that regard, at least. I'm not sure if this helps in your case, but we also have dependency license verification happening through CI on all pushes to the project to ensure that all dependencies are at least compatible with the apache2 license that Kibana itself is offered under.

I don't mind doing a little work to benefit from a great product, but Elasticsearch and Kibana don't build or test cleanly due to the dynamic nature of these package eco-systems. I found little support for building these products after reading through CONTRIBUTING.md, included docs, web searches, Stack Exchange, and discuss.elastic.co.

I'm really sorry that you experienced so much trouble with this. One of the main reasons why you haven't found good documentation anywhere on the web is that it's a precise-enough set of constraints, so you're finding yourself in a situation that not many other people have encountered. Uncommon or not though, we always want to improve our documentation and tooling to help folks work with Kibana, so please don't hesitate to open requests for enhancements to product, the build tools, or the docs. We handle all of these things through the repo on github.

1 Like
3

First off, thanks Court for the quick and to the point reply!

The offline ability to build Kibana can't be that far away. I suspect I can archive the node_modules directory in its present state. It has package names, versions, and licenses included. If this is actually viable, then I should be able to copy the npm packages from the node_modules directory of the officially published rpm/deb binary package to get the exact same versions of these modules.

Given that, the next 'online' challenges are:

  1. Preventing the need to access git/github.
  2. Caching whatever is done by downloadNodeBuilds

I suspect someone more familiar with the build environment (grunt?) would know how to modify the 'makefiles' so it skips the step if the resulting file(s) are already there. If anyone reading can offer some suggestions, I'll try them out. In the meanwhile, I will hack around.

The default for the build task is to build all of the baseline artifacts for Kibana, which includes deb and rpm packages. The --skip-os-packages specifically disables the building of rpm/deb packages, which is why none were created in your example.

CONTRIBUTING.md explains the rest of building the rpm/deb packages. Another language (ruby) and more modules (pleaserun) to curate. :frowning:

We include a NOTICE.txt file in the root of the repository that includes all of the license information from all dependencies in the project...

I was able to find this file, but not at the root of the repository. I found it in the build directory. Nice Job and makes my job easier!

4

Was able to take the following step: (assuming I build kibana in /opt/kibana)

  1. edit /opt/kibana/tasks/build/index.js, comment out all steps starting with '_build:notice' with prepended '//'.
  2. npm run build
  3. copy /opt/kibana/build/kibana/node_modules to a cache
  4. edit /opt/kibana/tasks/build/index.js, restore to normal

Then, when I want to build a fresh Kibana, with no network needed.

  1. start with a clean repository with node binaries (see steps in later post on how to copy 'node binaries')
  2. mkdir -p /opt/kibana/build/kibana/node_modules
  3. copy node_modules from cache to this new directory
  4. edit /opt/kibana/tasks/build/index.js, comment out the following build steps with preceding '//':
    clean:build
    clean:target
    _build:downloadNodeBuilds
    _build:extractNodeBuilds
    _build:installNpmDeps
  5. npm run build -- --osPackages (I add '-t deb' for debian only)

So please, tell me what might be wrong with going about it this way?

What I didn't understand completely is what magic is by all the different build steps to the node_modules directory. Can someone explain this too, please?

5

It's worth emphasizing again that none of this is supported, which means the steps that you describe today could work right now but are not guaranteed to continue working in the future without first-class support for this in the product.

That said, one thing I don't see included is handling the nodejs binaries. The downloadNodeBuilds task downloads the relevant node binaries from the internet and caches them to the .node_binaries directory. The extractNodeBuilds task then makes sure the binaries are properly extracted (some of them come as a zip, if I recall), and copies them to the relevant build directories based on the individual architectures that are supported. You could follow a similar set of steps as you did for node_modules to reuse this cache locally if you desire.

We don't really do anything to node_modules in the build. The installNpmDeps task basically just runs npm install --production --no-optional in the build directory, and since we don't use any native modules in builds, this directory is the same on each supported architecture.

6

Thanks again Court

I am not node.js/grunt fluent, so I'm sharing in hopes that one of the experts following this forum will pick this effort up and make it supported. What I am is a curator of Open Source software and inability to build Kibana offline from source is our requirement before using this software, or walking away. I doubt I am the only organization to have requirements such as these.

Yes, I did actually have some steps to save those away too.

  1. mkdir -p /opt/kibana/.node_binaries/6.11.1/linux-x64
  2. cp /node-v6.11.1-linux-x64.tar.gz /opt/kibana/.node_binaries/6.11.1
  3. cd /opt/kibana/.node_binaries/6.11.1
  4. tar -xf node-v6.11.1-linux-x64.tar.gz -C linux-x64 --strip-components=1
  5. export PATH=$PATH:/opt/kibana/.node_binaries/6.11.1/linux-x64/bin

I have other steps for copying Ruby packages (gems) for fpm and pleaseask into /var/lib/gems.

I still expect something is being done, albeit in a different step than what you pointed out. When the build happens with my steps, it works, but when I turn around and repeat the steps, it then warns of some kind of "de-optimization because size is greater than 500KB" step early on, then runs out of heap space.

I can't say this enough, this is a painful, confusing, and less-than-ideally documented build process. It scare many away from using this awesome product.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.