"View Metrics" shows GraphQL Error

[pfa]

Hi!

After migration from 7.1.1 to 7.3.0 the Infrastructure UI "Show Metrics" isn't working any more.
The tiles in global view(hosts and kubernetes) show correct metrics, "view logs" is also working properly.

Error Message: "GraphQL Error [object Object]"

i enabled debug mode for kibana, but didn't find any hint about it.

in 7.1.1 everything works as expected

[pfa]

today i installed 7.3.1. - still same behaviour.
found that metrics explorer also isn't working - here are some logs from kibana to metrics explorer

Aug 26 16:33:29 lnx0158a kibana[29798]: {"type":"log","@timestamp":"2019-08-26T14:33:29Z","tags":["license","debug","xpack"],"pid":29798,"message":"Calling [data] Elasticsearch _xpack API. Polling frequency: 30001"}
Aug 26 16:33:29 lnx0158a kibana[29798]: Debug: internal, implementation, error
Aug 26 16:33:29 lnx0158a kibana[29798]: AssertionError [ERR_ASSERTION]: Cannot wrap non-Error object
Aug 26 16:33:29 lnx0158a kibana[29798]: at new AssertionError (internal/assert.js:269:11)
Aug 26 16:33:29 lnx0158a kibana[29798]: at Object.exports.assert (/usr/share/kibana/x-pack/legacy/plugins/infra/node_modules/hoek/lib/index.js:524:11)
Aug 26 16:33:29 lnx0158a kibana[29798]: at boomify (/usr/share/kibana/x-pack/legacy/plugins/infra/node_modules/boom/lib/index.js:107:14)
Aug 26 16:33:29 lnx0158a kibana[29798]: at Object.handler (/usr/share/kibana/x-pack/legacy/plugins/infra/server/routes/metrics_explorer/index.js:50:15)
Aug 26 16:33:29 lnx0158a kibana[29798]: at process._tickCallback (internal/process/next_tick.js:68:7)
Aug 26 16:33:29 lnx0158a kibana[29798]: {"type":"error","@timestamp":"2019-08-26T14:33:29Z","tags":,"pid":29798,"level":"error","error":{"message":"Cannot wrap non-Error object","name":"AssertionError [ERR_ASSERTION]","stack":"AssertionError [ERR_ASSERTION]: Cannot wrap non-Error object\n at new AssertionError (internal/assert.js:269:11)\n at Object.exports.assert (/usr/share/kibana/x-pack/legacy/plugins/infra/node_modules/hoek/lib/index.js:524:11)\n at boomify (/usr/share/kibana/x-pack/legacy/plugins/infra/node_modules/boom/lib/index.js:107:14)\n at Object.handler (/usr/share/kibana/x-pack/legacy/plugins/infra/server/routes/metrics_explorer/index.js:50:15)\n at process._tickCallback (internal/process/next_tick.js:68:7)","code":"ERR_ASSERTION"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":null,"query":{},"pathname":"/api/infra/metrics_explorer","path":"/api/infra/metrics_explorer","href":"/api/infra/metrics_explorer"},"message":"Cannot wrap non-Error object"}

[simianhacker (Chris Cowan)]

The error in Kibana is from a bad dependency & bug (which I seem to remember fixing recently). There should be an error message in the Elasticsearch logs that will give us more insights into your issue.

If I had to make a prediction about what you're going to find in the ES logs, it will probably be some message about fielddata which means your index template for Metricbeat (mappings) are most likely not setup properly.

My guess... when the data was migrated the mappings didn't get updated to the most recent Metricbeat version. Did you also upgrade your Metricbeat agents?

The index templates (mapping) are tied directly to the version metricbeat-7.1.1-* which means any of the new data from the new version isn't applying the index template (mappings). The fix should be to re-run the setup step for Metricbeat (https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-template.html).

Let me know if that was the issue (and if the fix worked) or not. We can always troubleshoot it some more.

[pfa]

hi!

true, with the update from 7.3.0 to 7.3.1 i reload 7.3.1 templates from scratch and forgot to add my fielddata: true to [host.name] and [event.dataset].
after re-apply them i'm back to the initial "GraphQL error", for which the issue initial was openened

[simianhacker (Chris Cowan)]

@pfa Are there any errors in ES when the GraphQL error happens? Can you open the developer console in your browser, click on the network tab and inspect the GraphQL response with the error (you might have to click a few)?

[pfa]

hi,

no, no errors in ES at the time when Inrastructure UI is beeing used (there are a few, but related to other things.
also no error in Browser:

[simianhacker (Chris Cowan)]

For the GraphQL endpoints, you have to look at the responses to see the errors; they always return 200 (sigh).

[pfa]

sure, sorry:
don't know how to export result, so i took a screenshot:

[pfa]

Last Reply from graphlql is: {"data":null,"errors":[{"message":"[object Object]","locations":[{"line":4,"column":5}],"path":["source","metrics"]}]}

@simianhacker - is the content of the screenshot the output you expect from me? do you have any hint where the Problem can be?

thank you