Debugging missing plugins after updating from 7.6.2 to 7.9.2

tlacuache · October 16, 2020, 1:43pm

I'm working on updating a project from using Elastic OSS components at version 7.6.2 to version 7.9.2. For the most part everything has gone smoothly. However, I've gotten stuck on a few plugins that, although they install correctly, and don't throw any obvious errors per se, they don't seem to show up at all in the Kibana GUI.

I know the purpose of this forum is not to debug 3rd party plugins, and I'm not asking for that specifically. But I'm hoping for some direction on how I can debug what it is about the plugins that don't show up in the GUI vs. those that do so I can get it resolved.

Here are a list of the plugins I'm using:

As you can see in my Dockerfile, I'm swapping out the version strings from the released versions to 7.9.2 and making a few other tweaks that I've found are required to get the installation to work. During my docker image build, the plugins appear to succeed:

Installing ElastAlert plugin...
Archive:  elastalert-kibana-plugin.zip
  inflating: kibana/elastalert-kibana-plugin/package.json  
  inflating: kibana/elastalert-kibana-plugin/public/components/main/main.js  
updating: kibana/elastalert-kibana-plugin/package.json (deflated 42%)
updating: kibana/elastalert-kibana-plugin/public/components/main/main.js (deflated 63%)
Attempting to transfer from file:///tmp/elastalert-kibana-plugin.zip
Transferring 24641559 bytes....................
Transfer complete
Retrieving metadata from plugin archive
Extracting plugin archive
Extraction complete
Plugin installation complete


Installing Sankey visualization...
Archive:  /tmp/kibana-sankey.zip
...
Attempting to transfer from file:///tmp/sankey_vis.zip
Transferring 12062304 bytes....................
Transfer complete
Retrieving metadata from plugin archive
Extracting plugin archive
Extraction complete
Plugin installation complete


Installing Drilldown menu plugin...
Archive:  /tmp/kibana-drilldown.zip
...
Attempting to transfer from file:///tmp/drilldown.zip
Transferring 195325 bytes....................
Transfer complete
Retrieving metadata from plugin archive
Extracting plugin archive
Extraction complete
Plugin installation complete

Installing Comments visualization...
Archive:  kibana-comments.zip
  inflating: kibana/kibana-comments-app-plugin/package.json  
  inflating: kibana/kibana-comments-app-plugin/public/app.js  
updating: kibana/kibana-comments-app-plugin/package.json (deflated 48%)
updating: kibana/kibana-comments-app-plugin/public/app.js (deflated 75%)
Attempting to transfer from file:///tmp/kibana-comments.zip
Transferring 23644018 bytes....................
Transfer complete
Retrieving metadata from plugin archive
Extracting plugin archive
Extraction complete
Plugin installation complete

Installing Swimlanes visualization...
Archive:  kibana-swimlane.zip
  inflating: kibana/prelert_swimlane_vis/package.json  
updating: kibana/prelert_swimlane_vis/package.json (deflated 40%)
Attempting to transfer from file:///tmp/kibana-swimlane.zip
Transferring 251923 bytes....................
Transfer complete
Retrieving metadata from plugin archive
Extracting plugin archive
Extraction complete
Plugin installation complete

Here are the relevant lines from the logs at Kibana startup:

2020-10-16T13:12:44Z info plugins-service Plugin "visTypeXy" is disabled. 
2020-10-16T13:12:46Z warning legacy-service Some installed third party plugin(s) [elastalert-kibana-plugin, kbn_sankey_vis, kibana-comments-app-plugin, kibana-plugin-drilldownmenu, prelert_swimlane_vis] are using the legacy plugin format and will no longer work in a future Kibana release. Please refer to https://ela.st/kibana-breaking-changes-8-0 for a list of breaking changes and https://ela.st/kibana-platform-migration for documentation on how to migrate legacy plugins. 
2020-10-16T13:12:46Z warning config","deprecation kibana.defaultAppId is deprecated and will be removed in 8.0. Please use the `defaultRoute` advanced setting instead 
2020-10-16T13:12:46Z info plugins-system Setting up [38] plugins: [usageCollection,telemetryCollectionManager,telemetry,kibanaUsageCollection,newsfeed,mapsLegacy,kibanaLegacy,timelion,share,legacyExport,esUiShared,charts,bfetch,expressions,data,home,console,apmOss,management,indexPatternManagement,advancedSettings,savedObjects,visualizations,visualize,visTypeVislib,visTypeVega,visTypeTimeseries,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeMarkdown,tileMap,regionMap,inputControlVis,discover,dashboard,savedObjectsManagement] 
2020-10-16T13:12:46Z info savedobjects-service Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations... 
2020-10-16T13:12:46Z info savedobjects-service Starting saved objects migrations 
2020-10-16T13:12:46Z info savedobjects-service Creating index .kibana_1. 
2020-10-16T13:12:47Z info savedobjects-service Pointing alias .kibana to .kibana_1. 
2020-10-16T13:12:47Z info savedobjects-service Finished in 694ms. 
2020-10-16T13:12:47Z info plugins-system Starting [38] plugins: [usageCollection,telemetryCollectionManager,telemetry,kibanaUsageCollection,newsfeed,mapsLegacy,kibanaLegacy,timelion,share,legacyExport,esUiShared,charts,bfetch,expressions,data,home,console,apmOss,management,indexPatternManagement,advancedSettings,savedObjects,visualizations,visualize,visTypeVislib,visTypeVega,visTypeTimeseries,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeMarkdown,tileMap,regionMap,inputControlVis,discover,dashboard,savedObjectsManagement] 
2020-10-16T13:12:49Z info optimize Optimizing and caching bundles for elastalert-kibana-plugin, kibana-comments-app-plugin, status_page and timelion. This may take a few minutes 
                          Error in worker TypeError [ERR_INVALID_ARG_TYPE]: The "id" argument must be of type string. Received type object
                          Error in worker TypeError [ERR_INVALID_ARG_TYPE]: The "id" argument must be of type string. Received type object
                          Error in worker TypeError [ERR_INVALID_ARG_TYPE]: The "id" argument must be of type string. Received type object
2020-10-16T13:13:41Z info optimize Optimization of bundles for elastalert-kibana-plugin, kibana-comments-app-plugin, status_page and timelion complete in 51.78 seconds 
2020-10-16T13:13:41Z status plugin:kibana@7.9.2 Status changed from uninitialized to green - Ready
...
2020-10-16T13:13:41Z status plugin:elasticsearch@7.9.2 Status changed from yellow to green - Ready
2020-10-16T13:13:41Z status plugin:elastalert-kibana-plugin@1.3.0 Status changed from uninitialized to green - Ready
2020-10-16T13:13:41Z status plugin:kibana-comments-app-plugin@7.9.2-1 Status changed from uninitialized to green - Ready
2020-10-16T13:13:41Z info Server running at http://0:5601/kibana 
2020-10-16T13:13:41Z info Kibana http server running at http://0:5601/kibana

I notice a few things that may(?) be related:

The "Some installed third party plugins are using the legacy plugin format" message: I understand that they'll be deprecated in a future release, and I'll have to deal with that at that point. But 7.9.2 does still support the legacy plugin format, right? You'll notice that the list of plugins here includes both the working and "broken" plugins I'm installing.
The "Optimization of bundles for elastalert-kibana-plugin, kibana-comments-app-plugin..." message: this is the most revealing one, probably, as it shows of the five plugins, the only two that are optimizing happen to be the ones that work

The other three plugins all add a visualization type to be used in visualizations/dashboards. However, in the "new visualization" UI their icons don't show up at all.

I'm not really a Kibana plugin developer, I know just enough to poke and prod at it to get it up and running. But I'm at a bit of a loss here and am looking for some directions or help to know how I can debug why these plugins aren't showing up.

Thanks!

Mikhail_Shustov · October 20, 2020, 7:42am

Hi.

In the Kibana legacy plugin system, every plugin version should be compatible with the Kibana version.
All the missing plugins declare that they have been built for v7.6.2/7.6.3, so Kibana disabled them.

github.com

prelert/kibana-swimlane-vis/blob/master/package.json#8

{
  "name": "prelert_swimlane_vis",
  "version": "7.6.2",
  "description": "Prelert swimlane visualization plugin",
  "license": "Apache-2.0",
  "main": "index.js",
  "kibana": {
    "version": "7.6.2"
  },
  "scripts": {
    "preinstall": "node ../../preinstall_check",
    "kbn": "node ../../scripts/kbn",
    "es": "node ../../scripts/es",
    "lint": "eslint .",
    "start": "plugin-helpers start",
    "test:server": "plugin-helpers test:server",
    "test:browser": "plugin-helpers test:browser",
    "build": "plugin-helpers build"
  },
  "devDependencies": {

This file has been truncated. show original

github.com

mmguero-dev/kbn_sankey_vis/blob/master/package.json

{
  "name": "kbn_sankey_vis",
  "version": "7.6.3",
  "authors": [
    "Chenryn <rao.chenlin@gmail.com>",
    "Ch-bas <bassem.chagra@uniberg.com>"
  ],
  "description": "Sankey Diagram Visualization Plugin",
  "license": "Apache-2.0",
  "homepage": "https://github.com/uniberg/kbn_sankey_vis",
  "repository": "https://github.com/uniberg/kbn_sankey_vis.git",
  "main": "index.js",
  "scripts": {
    "test": "nyc --all mocha"
  },
  "dependencies": {
    "d3-plugins-sankey": "^1.2.1",
    "json-stable-stringify": "^1.0.1"
  },
  "devDependencies": {

This file has been truncated. show original

github.com

mmguero-dev/kibana-plugin-drilldownmenu/blob/master/package.json#L6


{
  "name": "kibana-plugin-drilldownmenu",
  "version": "1.0.0",
  "description": "A field-formatter plugin allowing fields to be drilled down into multiple visualizations/dashboards",
  "kibana": {
    "version": "7.6.2"
  }
}

Plugins need to be updated to be compatible with v7.9.2. It might be not possible since Kibana is migrating to the new plugin system https://www.elastic.co/blog/introducing-a-new-architecture-for-kibana
So the best you can do:

help them to migrate to the Kibana platform. https://github.com/elastic/kibana/blob/master/src/core/MIGRATION.md
downgrade your Kibana version to v7.6.2 until all the dependencies migrate to the Kibana platform

tlacuache · October 20, 2020, 1:11pm

Hi, thanks for responding!

I am familiar with the requirement that the versions match. I don't think that, at least in this case, this is what's happening. If you look at my build process in the Dockerfile you will see something like this for each plugin:

    echo "Installing Swimlanes visualization..." && \
      unzip kibana-swimlane.zip kibana/prelert_swimlane_vis/package.json && \
      sed -i "s/7\.6\.2/7\.9\.2/g" kibana/prelert_swimlane_vis/package.json && \
      zip kibana-swimlane.zip kibana/prelert_swimlane_vis/package.json && \
      cd /usr/share/kibana/plugins && \
      /usr/share/kibana/bin/kibana-plugin install file:///tmp/kibana-swimlane.zip --allow-root

The unzip, sed and zip commands are being used to modify the package.json file of each of the plugins and adjust the version to 7.9.2 so that the installation proceeds. I'm aware that there may be other incompatibilities that I need to address (obviously there are, which is what I'm trying to figure out with this post) but it's not just as simple as "the versions don't match" because I am already addressing that. Additionally, if the issue was that the versions don't match Kibana does (at least it has for me in the past) print out a different error message specifically highlighting the version issue, which I am not seeing.

Mikhail_Shustov · October 21, 2020, 1:49pm

All 3 plugins add custom functionality to Kibana via visTypes extension point

The problem that visualization plugin migrated to the new Kibana platform in v7.8.0 move visualizations plugin to new platform by ppisljar · Pull Request #60403 · elastic/kibana · GitHub and doesn't use uiExports.visTypes anymore.

You can use Kibana with version below v7.8 or adjust plugins code with one of 2 options:

migrate plugins to Kibana platform. https://github.com/elastic/kibana/blob/master/src/core/MIGRATION.md
use a compatibility layer

in hacks.js
import { npSetup } from 'ui/new_platform';
npSetup.plugins. visualizations.createBaseVisualization(...)

Probably it's better to migrate to KP because we removed legacy platform plugins support in v7.10

tlacuache · October 21, 2020, 3:28pm

Thank you for the direction. I really appreciate your time.

tlacuache · October 21, 2020, 4:57pm

This is the real drawback about using "community"/third-party plugins, isn't it? Compatibility changes and suddenly you're up the creek.

Mikhail_Shustov · October 22, 2020, 7:56am

Yes, but we aim to stabilize the Kibana platform API with the new platform architecture. Of course, it will take some time.

system · November 19, 2020, 7:56am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.