[kibana 6.5.1] Reporting fails with elasticcloud endpoint (but not elasticsearch selfhosted)

dao · December 31, 2018, 12:31pm

Hello,

I am connecting my self hosted kibana to an elastic cloud endpoint. I know that elastic cloud is shipped with a kibana, but for some reasons, I have to use my own kibana. Unfortunately, the reporting keeps failing with:

Here is my configuration. Can you help me? If i find the fix, I will be able to migrate to the elastic cloud

root@int:~# cat /etc/kibana/kibana.yml | grep -v ^# | grep -v ^$
elasticsearch.url: "https://44dCLOUDN4dd3c012baa1dc4.eu-west-1.aws.found.io:9243"
kibana.index: .kibana-int
elasticsearch.username: "kibana-wilco"
elasticsearch.password: "password"
logging.dest: /var/log/kibana/kibana.log
logging.verbose: true
xpack.reporting.encryptionKey: "ZHDFEBHAIJOSLNKJZBCKHJESMO682351279"
xpack.reporting.kibanaServer.port: 443
xpack.reporting.kibanaServer.protocol: https
xpack.reporting.kibanaServer.hostname: "int.flightwatching.com"
xpack.reporting.index: ".reporting-int"

Larry_Gregory · December 31, 2018, 4:44pm

Hey @dao,

I see you have verbose logging enabled. Can you post an excerpt from your kibana logs when the reporting job fails? Kibana should be writing errors to the log indicating what happened.

dao · December 31, 2018, 5:18pm

Hello Larry,

here are the logs. I have the har (from my browser, but it is too large for the discuss)

(Attachment Log kibana int.flightwatching.com.txt is missing)

Larry_Gregory · December 31, 2018, 6:00pm

Thanks Olivier -- it looks like the Kibana log failed to upload for whatever reason. Can you retry?

dao · December 31, 2018, 6:16pm

Keeps failing the same way, it is the case for a long time:

dao · January 2, 2019, 9:06am

Hello Larry, any clue?

Larry_Gregory · January 2, 2019, 1:00pm

Without the Kibana server logs, it's really tricky to debug this any further. Can you try following the troubleshooting steps at the bottom of this page for your particular error message? https://www.elastic.co/guide/en/kibana/master/reporting-troubleshooting.html#_literal_caught_error_spawning_chromium_literal

dao · January 2, 2019, 1:53pm

Hello Larry,

I tried to follow the procedure for troubleshoot, and it seems that the script is OK:

root@int:~# ls -latr /var/lib/kibana/headless_shell-linux/

total 111124

drwxrwxr-x 3 kibana kibana 4096 Dec 12 16:43 ..

drwxr-xr-x 2 kibana kibana 4096 Dec 12 16:43 .

-rwxr-xr-x 1 kibana kibana 113780392 Dec 12 16:43 headless_shell

root@int:~# ps -aef | grep kibana

kibana 7072 1 1 2018 ? 06:46:19 /usr/share/kibana/bin/../node/bin/node --no-warnings /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

dao · January 2, 2019, 1:48pm

Hello Larry,

I thought I sent the logs to you. But it seems that the post failed because they are too large.

How can I send them to you?

Larry_Gregory · January 2, 2019, 2:14pm

I don't need the full log file, but if there are relevant portions of the log when Kibana tries to generate the report, that would be useful to post.

dao · January 2, 2019, 2:41pm

here is the request (to be continued):

{"type":"log","@timestamp":"2018-12-17T08:13:18Z","tags":["trace","legacy","service"],"pid":7072,"message":"Request will be handled by proxy POST:/api/reporting/generate/printablePdf?jobParams=(browserTimezone%3AEurope%2FParis%2Clayout%3A(dimensions%3A(height%3A555.234375%2Cwidth%3A1129)%2Cid%3Apreserve_layout)%2CobjectType%3Avisualization%2CrelativeUrls%3A!(%27%2Fapp%2Fkibana%23%2Fvisualize%2Fedit%2Fc27f9c10-fe28-11e8-9e04-ed8c3381d671%3F_g%3D()%26_a%3D(filters%3A!!()%2Clinked%3A!!f%2Cquery%3A(language%3Alucene%2Cquery%3A!%27!%27)%2CuiState%3A(vis%3A(defaultColors%3A(!%270%2B-%2B50!%27%3A!%27rgb(0%2C104%2C55)!%27%2C!%2750%2B-%2B75!%27%3A!%27rgb(255%2C255%2C190)!%27%2C!%2775%2B-%2B100!%27%3A!%27rgb(165%2C0%2C38)!%27)))%2Cvis%3A(aggs%3A!!((enabled%3A!!t%2Cid%3A!%271!%27%2Cparams%3A()%2Cschema%3Ametric%2Ctype%3Acount))%2Cparams%3A(addLegend%3A!!t%2CaddTooltip%3A!!t%2Cgauge%3A(backStyle%3AFull%2CcolorSchema%3A!%27Green%2Bto%2BRed!%27%2CcolorsRange%3A!!((from%3A0%2Cto%3A50)%2C(from%3A50%2Cto%3A75)%2C(from%3A75%2Cto%3A100))%2CextendRange%3A!!t%2CgaugeColorMode%3ALabels%2CgaugeStyle%3AFull%2CgaugeType%3AArc%2CinvertColors%3A!!f%2Clabels%3A(color%3Ablack%2Cshow%3A!!t)%2Corientation%3Avertical%2CpercentageMode%3A!!f%2Cscale%3A(color%3A%2523333%2Clabels%3A!!f%2Cshow%3A!!t)%2Cstyle%3A(bgColor%3A!!f%2CbgFill%3A%2523eee%2CbgMask%3A!!f%2CbgWidth%3A0.9%2CfontSize%3A60%2ClabelColor%3A!!t%2Cmask%3A!!f%2CmaskBars%3A50%2CsubText%3A!%27!%27%2Cwidth%3A0.9)%2Ctype%3Ameter%2CverticalSplit%3A!!f)%2CisDisplayWarning%3A!!f%2Ctype%3Agauge)%2Ctitle%3A!%27New%2BVisualization!%27%2Ctype%3Agauge))%27)%2Ctitle%3A%27New%20Visualization%27)."}

{"type":"log","@timestamp":"2018-12-17T08:13:18Z","tags":["debug","security","basic"],"pid":7072,"message":"Trying to authenticate user request to /api/reporting/generate/printablePdf?jobParams=(browserTimezone%3AEurope%2FParis%2Clayout%3A(dimensions%3A(height%3A555.234375%2Cwidth%3A1129)%2Cid%3Apreserve_layout)%2CobjectType%3Avisualization%2CrelativeUrls%3A!(%27%2Fapp%2Fkibana%23%2Fvisualize%2Fedit%2Fc27f9c10-fe28-11e8-9e04-ed8c3381d671%3F_g%3D()%26_a%3D(filters%3A!!()%2Clinked%3A!!f%2Cquery%3A(language%3Alucene%2Cquery%3A!%27!%27)%2CuiState%3A(vis%3A(defaultColors%3A(!%270%2B-%2B50!%27%3A!%27rgb(0%2C104%2C55)!%27%2C!%2750%2B-%2B75!%27%3A!%27rgb(255%2C255%2C190)!%27%2C!%2775%2B-%2B100!%27%3A!%27rgb(165%2C0%2C38)!%27)))%2Cvis%3A(aggs%3A!!((enabled%3A!!t%2Cid%3A!%271!%27%2Cparams%3A()%2Cschema%3Ametric%2Ctype%3Acount))%2Cparams%3A(addLegend%3A!!t%2CaddTooltip%3A!!t%2Cgauge%3A(backStyle%3AFull%2CcolorSchema%3A!%27Green%2Bto%2BRed!%27%2CcolorsRange%3A!!((from%3A0%2Cto%3A50)%2C(from%3A50%2Cto%3A75)%2C(from%3A75%2Cto%3A100))%2CextendRange%3A!!t%2CgaugeColorMode%3ALabels%2CgaugeStyle%3AFull%2CgaugeType%3AArc%2CinvertColors%3A!!f%2Clabels%3A(color%3Ablack%2Cshow%3A!!t)%2Corientation%3Avertical%2CpercentageMode%3A!!f%2Cscale%3A(color%3A%2523333%2Clabels%3A!!f%2Cshow%3A!!t)%2Cstyle%3A(bgColor%3A!!f%2CbgFill%3A%2523eee%2CbgMask%3A!!f%2CbgWidth%3A0.9%2CfontSize%3A60%2ClabelColor%3A!!t%2Cmask%3A!!f%2CmaskBars%3A50%2CsubText%3A!%27!%27%2Cwidth%3A0.9)%2Ctype%3Ameter%2CverticalSplit%3A!!f)%2CisDisplayWarning%3A!!f%2Ctype%3Agauge)%2Ctitle%3A!%27New%2BVisualization!%27%2Ctype%3Agauge))%27)%2Ctitle%3A%27New%20Visualization%27)."}

{"type":"log","@timestamp":"2018-12-17T08:13:18Z","tags":["debug","security","basic"],"pid":7072,"message":"Trying to authenticate via header."}

{"type":"log","@timestamp":"2018-12-17T08:13:18Z","tags":["debug","security","basic"],"pid":7072,"message":"Authorization header is not presented."}

{"type":"log","@timestamp":"2018-12-17T08:13:18Z","tags":["debug","security","basic"],"pid":7072,"message":"Trying to authenticate via state."}

{"type":"log","@timestamp":"2018-12-17T08:13:18Z","tags":["plugin","debug"],"pid":7072,"message":"Checking Elasticsearch version"}

{"type":"log","@timestamp":"2018-12-17T08:13:18Z","tags":["debug","security","basic"],"pid":7072,"message":"Request has been authenticated via state."}

{"type":"log","@timestamp":"2018-12-17T08:13:18Z","tags":["reporting","esqueue","job","debug"],"pid":7072,"message":"jps1l7oh05ggf4f1af3pnefc - Job created in index .reporting-int-2018.12.16"}

{"type":"log","@timestamp":"2018-12-17T08:13:18Z","tags":["reporting","esqueue","job","debug"],"pid":7072,"message":"jps1l7oh05ggf4f1af3pnefc - Job index refreshed .reporting-int-2018.12.16"}

{"type":"log","@timestamp":"2018-12-17T08:13:18Z","tags":["reporting","debug"],"pid":7072,"message":"Saved object to process"}

dao · January 2, 2019, 2:42pm

And the rest up to the error:

{"type":"response","@timestamp":"2018-12-17T08:13:18Z","tags":,"pid":7072,"method":"post","statusCode":200,"req":{"url":"/api/reporting/generate/printablePdf?jobParams=(browserTimezone%3AEurope%2FParis%2Clayout%3A(dimensions%3A(height%3A555.234375%2Cwidth%3A1129)%2Cid%3Apreserve_layout)%2CobjectType%3Avisualization%2CrelativeUrls%3A!('%2Fapp%2Fkibana%23%2Fvisualize%2Fedit%2Fc27f9c10-fe28-11e8-9e04-ed8c3381d671%3F_g%3D()%26_a%3D(filters%3A!!()%2Clinked%3A!!f%2Cquery%3A(language%3Alucene%2Cquery%3A!'!')%2CuiState%3A(vis%3A(defaultColors%3A(!'0%2B-%2B50!'%3A!'rgb(0%2C104%2C55)!'%2C!'50%2B-%2B75!'%3A!'rgb(255%2C255%2C190)!'%2C!'75%2B-%2B100!'%3A!'rgb(165%2C0%2C38)!')))%2Cvis%3A(aggs%3A!!((enabled%3A!!t%2Cid%3A!'1!'%2Cparams%3A()%2Cschema%3Ametric%2Ctype%3Acount))%2Cparams%3A(addLegend%3A!!t%2CaddTooltip%3A!!t%2Cgauge%3A(backStyle%3AFull%2CcolorSchema%3A!'Green%2Bto%2BRed!'%2CcolorsRange%3A!!((from%3A0%2Cto%3A50)%2C(from%3A50%2Cto%3A75)%2C(from%3A75%2Cto%3A100))%2CextendRange%3A!!t%2CgaugeColorMode%3ALabels%2CgaugeStyle%3AFull%2CgaugeType%3AArc%2CinvertColors%3A!!f%2Clabels%3A(color%3Ablack%2Cshow%3A!!t)%2Corientation%3Avertical%2CpercentageMode%3A!!f%2Cscale%3A(color%3A%2523333%2Clabels%3A!!f%2Cshow%3A!!t)%2Cstyle%3A(bgColor%3A!!f%2CbgFill%3A%2523eee%2CbgMask%3A!!f%2CbgWidth%3A0.9%2CfontSize%3A60%2ClabelColor%3A!!t%2Cmask%3A!!f%2CmaskBars%3A50%2CsubText%3A!'!'%2Cwidth%3A0.9)%2Ctype%3Ameter%2CverticalSplit%3A!!f)%2CisDisplayWarning%3A!!f%2Ctype%3Agauge)%2Ctitle%3A!'New%2BVisualization!'%2Ctype%3Agauge))')%2Ctitle%3A'New%20Visualization')","method":"post","headers":{"connection":"upgrade","host":"int.flightwatching.com","content-length":"0","origin":"https://int.flightwatching.com","kbn-version":"6.5.1","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36","content-type":"application/json","accept":"/","referer":"https://int.flightwatching.com/app/kibana","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,fr;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"https://int.flightwatching.com/app/kibana"},"res":{"statusCode":200,"responseTime":349,"contentLength":9},"message":"POST /api/reporting/generate/printablePdf?jobParams=(browserTimezone%3AEurope%2FParis%2Clayout%3A(dimensions%3A(height%3A555.234375%2Cwidth%3A1129)%2Cid%3Apreserve_layout)%2CobjectType%3Avisualization%2CrelativeUrls%3A!('%2Fapp%2Fkibana%23%2Fvisualize%2Fedit%2Fc27f9c10-fe28-11e8-9e04-ed8c3381d671%3F_g%3D()%26_a%3D(filters%3A!!()%2Clinked%3A!!f%2Cquery%3A(language%3Alucene%2Cquery%3A!'!')%2CuiState%3A(vis%3A(defaultColors%3A(!'0%2B-%2B50!'%3A!'rgb(0%2C104%2C55)!'%2C!'50%2B-%2B75!'%3A!'rgb(255%2C255%2C190)!'%2C!'75%2B-%2B100!'%3A!'rgb(165%2C0%2C38)!')))%2Cvis%3A(aggs%3A!!((enabled%3A!!t%2Cid%3A!'1!'%2Cparams%3A()%2Cschema%3Ametric%2Ctype%3Acount))%2Cparams%3A(addLegend%3A!!t%2CaddTooltip%3A!!t%2Cgauge%3A(backStyle%3AFull%2CcolorSchema%3A!'Green%2Bto%2BRed!'%2CcolorsRange%3A!!((from%3A0%2Cto%3A50)%2C(from%3A50%2Cto%3A75)%2C(from%3A75%2Cto%3A100))%2CextendRange%3A!!t%2CgaugeColorMode%3ALabels%2CgaugeStyle%3AFull%2CgaugeType%3AArc%2CinvertColors%3A!!f%2Clabels%3A(color%3Ablack%2Cshow%3A!!t)%2Corientation%3Avertical%2CpercentageMode%3A!!f%2Cscale%3A(color%3A%2523333%2Clabels%3A!!f%2Cshow%3A!!t)%2Cstyle%3A(bgColor%3A!!f%2CbgFill%3A%2523eee%2CbgMask%3A!!f%2CbgWidth%3A0.9%2CfontSize%3A60%2ClabelColor%3A!!t%2Cmask%3A!!f%2CmaskBars%3A50%2CsubText%3A!'!'%2Cwidth%3A0.9)%2Ctype%3Ameter%2CverticalSplit%3A!!f)%2CisDisplayWarning%3A!!f%2Ctype%3Agauge)%2Ctitle%3A!'New%2BVisualization!'%2Ctype%3Agauge))')%2Ctitle%3A'New%20Visualization') 200 349ms - 9.0B"}

{"type":"log","@timestamp":"2018-12-17T08:13:19Z","tags":["reporting","esqueue","worker","debug"],"pid":7072,"message":"jps1iru105ggf4f1af688tyb - 1 outstanding jobs returned"}

{"type":"log","@timestamp":"2018-12-17T08:13:19Z","tags":["license","debug","xpack"],"pid":7072,"message":"Calling [data] Elasticsearch _xpack API. Polling frequency: 30001"}

{"type":"log","@timestamp":"2018-12-17T08:13:19Z","tags":["reporting","esqueue","worker","debug"],"pid":7072,"message":"jps1iru105ggf4f1af688tyb - Claimed job jps1l7oh05ggf4f1af3pnefc"}

{"type":"log","@timestamp":"2018-12-17T08:13:19Z","tags":["debug","legacy","proxy"],"pid":7072,"message":"Event is being forwarded: connection"}

{"type":"log","@timestamp":"2018-12-17T08:13:19Z","tags":["reporting","esqueue","worker","debug"],"pid":7072,"message":"jps1iru105ggf4f1af688tyb - Starting job jps1l7oh05ggf4f1af3pnefc"}

{"type":"log","@timestamp":"2018-12-17T08:13:19Z","tags":["reporting","worker","debug"],"pid":7072,"message":"PDF: Processing PDF job"}

{"type":"log","@timestamp":"2018-12-17T08:13:19Z","tags":["trace","legacy","service"],"pid":7072,"message":"Request will be handled by proxy GET:/api/reporting/jobs/list?page=0&ids=jps1l7oh05ggf4f1af3pnefc."}

{"type":"log","@timestamp":"2018-12-17T08:13:19Z","tags":["debug","security","basic"],"pid":7072,"message":"Trying to authenticate user request to /api/reporting/jobs/list?page=0&ids=jps1l7oh05ggf4f1af3pnefc."}

{"type":"log","@timestamp":"2018-12-17T08:13:19Z","tags":["debug","security","basic"],"pid":7072,"message":"Trying to authenticate via header."}

{"type":"log","@timestamp":"2018-12-17T08:13:19Z","tags":["debug","security","basic"],"pid":7072,"message":"Authorization header is not presented."}

{"type":"log","@timestamp":"2018-12-17T08:13:19Z","tags":["debug","security","basic"],"pid":7072,"message":"Trying to authenticate via state."}

{"type":"log","@timestamp":"2018-12-17T08:13:20Z","tags":["debug","security","basic"],"pid":7072,"message":"Request has been authenticated via state."}

{"type":"log","@timestamp":"2018-12-17T08:13:20Z","tags":["reporting","screenshots","debug"],"pid":7072,"message":"Creating browser driver factory"}

{"type":"log","@timestamp":"2018-12-17T08:13:20Z","tags":["reporting","esqueue","worker","warn"],"pid":7072,"message":"jps1iru105ggf4f1af688tyb - Failure occurred on job jps1l7oh05ggf4f1af3pnefc: Error: Caught error spawning Chromium\n at Observable.Rx.Observable.create [as _subscribe] (/usr/share/kibana/node_modules/x-pack/plugins/reporting/server/browsers/chromium/driver_factory/index.js:56:24)\n at \n at process._tickDomainCallback (internal/process/next_tick.js:228:7)"}

joelgriffith · January 2, 2019, 9:17pm

There's been some recently discovered packages that might be missing in your operating-system (which would also be good to know for the purposes of diagnosing). You might try setting your log events to:

logging.events.logs: ['info', 'warning', 'error', 'fatal', 'reporting']

After that is done, run your report and see what this logfile contains:

/path/to/kibana/data/headless_shell-linux/chrome_debug.log

This is the dump of chromium's logs, and there might be an indicator here as to what the issue is. My current guess is there is a missing system package.

dao · January 4, 2019, 9:55am

Hello Joel,

I have set the logging conf as mentionned and retried the reporting. Still failing but no log file for chrome_debug.log...

Here is the log for kibana. We have a Creating browser driver factoryand then Failure occurred on job jqhuxi0q07gdf4f1afcl7uto: Error: Caught error spawning Chromium...

{"type":"response","@timestamp":"2019-01-04T09:48:54Z","tags":[],"pid":9661,"method":"post","statusCode":200,"req":{"url":"/api/reporting/generate/printablePdf?jobParams=(browserTimezone%3AEurope%2FParis%2Clayout%3A(dimensions%3A(height%3A1114.234375%2Cwidth%3A1358)%2Cid%3Apreserve_layout)%2CobjectType%3Avisualization%2CrelativeUrls%3A!('%2Fapp%2Fkibana%23%2Fvisualize%2Fedit%2Fc27f9c10-fe28-11e8-9e04-ed8c3381d671%3F_g%3D()%26_a%3D(filters%3A!!()%2Clinked%3A!!f%2Cquery%3A(language%3Alucene%2Cquery%3A!'!')%2CuiState%3A(vis%3A(defaultColors%3A(!'0%2B-%2B50!'%3A!'rgb(0%2C104%2C55)!'%2C!'50%2B-%2B75!'%3A!'rgb(255%2C255%2C190)!'%2C!'75%2B-%2B100!'%3A!'rgb(165%2C0%2C38)!')))%2Cvis%3A(aggs%3A!!((enabled%3A!!t%2Cid%3A!'1!'%2Cparams%3A()%2Cschema%3Ametric%2Ctype%3Acount))%2Cparams%3A(addLegend%3A!!t%2CaddTooltip%3A!!t%2Cgauge%3A(backStyle%3AFull%2CcolorSchema%3A!'Green%2Bto%2BRed!'%2CcolorsRange%3A!!((from%3A0%2Cto%3A50)%2C(from%3A50%2Cto%3A75)%2C(from%3A75%2Cto%3A100))%2CextendRange%3A!!t%2CgaugeColorMode%3ALabels%2CgaugeStyle%3AFull%2CgaugeType%3AArc%2CinvertColors%3A!!f%2Clabels%3A(color%3Ablack%2Cshow%3A!!t)%2Corientation%3Avertical%2CpercentageMode%3A!!f%2Cscale%3A(color%3A%2523333%2Clabels%3A!!f%2Cshow%3A!!t)%2Cstyle%3A(bgColor%3A!!f%2CbgFill%3A%2523eee%2CbgMask%3A!!f%2CbgWidth%3A0.9%2CfontSize%3A60%2ClabelColor%3A!!t%2Cmask%3A!!f%2CmaskBars%3A50%2CsubText%3A!'!'%2Cwidth%3A0.9)%2Ctype%3Ameter%2CverticalSplit%3A!!f)%2CisDisplayWarning%3A!!f%2Ctype%3Agauge)%2Ctitle%3A!'New%2BVisualization!'%2Ctype%3Agauge))')%2Ctitle%3A'New%20Visualization')","method":"post","headers":{"connection":"upgrade","host":"int.flightwatching.com","content-length":"0","origin":"https://int.flightwatching.com","kbn-version":"6.5.1","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36","content-type":"application/json","accept":"*/*","referer":"https://int.flightwatching.com/app/kibana","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,fr;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"https://int.flightwatching.com/app/kibana"},"res":{"statusCode":200,"responseTime":314,"contentLength":9},"message":"POST /api/reporting/generate/printablePdf?jobParams=(browserTimezone%3AEurope%2FParis%2Clayout%3A(dimensions%3A(height%3A1114.234375%2Cwidth%3A1358)%2Cid%3Apreserve_layout)%2CobjectType%3Avisualization%2CrelativeUrls%3A!('%2Fapp%2Fkibana%23%2Fvisualize%2Fedit%2Fc27f9c10-fe28-11e8-9e04-ed8c3381d671%3F_g%3D()%26_a%3D(filters%3A!!()%2Clinked%3A!!f%2Cquery%3A(language%3Alucene%2Cquery%3A!'!')%2CuiState%3A(vis%3A(defaultColors%3A(!'0%2B-%2B50!'%3A!'rgb(0%2C104%2C55)!'%2C!'50%2B-%2B75!'%3A!'rgb(255%2C255%2C190)!'%2C!'75%2B-%2B100!'%3A!'rgb(165%2C0%2C38)!')))%2Cvis%3A(aggs%3A!!((enabled%3A!!t%2Cid%3A!'1!'%2Cparams%3A()%2Cschema%3Ametric%2Ctype%3Acount))%2Cparams%3A(addLegend%3A!!t%2CaddTooltip%3A!!t%2Cgauge%3A(backStyle%3AFull%2CcolorSchema%3A!'Green%2Bto%2BRed!'%2CcolorsRange%3A!!((from%3A0%2Cto%3A50)%2C(from%3A50%2Cto%3A75)%2C(from%3A75%2Cto%3A100))%2CextendRange%3A!!t%2CgaugeColorMode%3ALabels%2CgaugeStyle%3AFull%2CgaugeType%3AArc%2CinvertColors%3A!!f%2Clabels%3A(color%3Ablack%2Cshow%3A!!t)%2Corientation%3Avertical%2CpercentageMode%3A!!f%2Cscale%3A(color%3A%2523333%2Clabels%3A!!f%2Cshow%3A!!t)%2Cstyle%3A(bgColor%3A!!f%2CbgFill%3A%2523eee%2CbgMask%3A!!f%2CbgWidth%3A0.9%2CfontSize%3A60%2ClabelColor%3A!!t%2Cmask%3A!!f%2CmaskBars%3A50%2CsubText%3A!'!'%2Cwidth%3A0.9)%2Ctype%3Ameter%2CverticalSplit%3A!!f)%2CisDisplayWarning%3A!!f%2Ctype%3Agauge)%2Ctitle%3A!'New%2BVisualization!'%2Ctype%3Agauge))')%2Ctitle%3A'New%20Visualization') 200 314ms - 9.0B"}
{"type":"log","@timestamp":"2019-01-04T09:48:55Z","tags":["plugin","debug"],"pid":9661,"message":"Checking Elasticsearch version"}
{"type":"log","@timestamp":"2019-01-04T09:48:55Z","tags":["reporting","esqueue","worker","debug"],"pid":9661,"message":"jqhuot0r07gdf4f1af1wekmp - 1 outstanding jobs returned"}
{"type":"log","@timestamp":"2019-01-04T09:48:55Z","tags":["reporting","esqueue","worker","debug"],"pid":9661,"message":"jqhuot0r07gdf4f1af1wekmp - Claimed job jqhuxi0q07gdf4f1afcl7uto"}
{"type":"log","@timestamp":"2019-01-04T09:48:55Z","tags":["reporting","esqueue","worker","debug"],"pid":9661,"message":"jqhuot0r07gdf4f1af1wekmp - Starting job jqhuxi0q07gdf4f1afcl7uto"}
{"type":"log","@timestamp":"2019-01-04T09:48:55Z","tags":["reporting","worker","debug"],"pid":9661,"message":"PDF: Processing PDF job"}
{"type":"log","@timestamp":"2019-01-04T09:48:55Z","tags":["reporting","screenshots","debug"],"pid":9661,"message":"Creating browser driver factory"}
{"type":"log","@timestamp":"2019-01-04T09:48:55Z","tags":["reporting","esqueue","worker","warn"],"pid":9661,"message":"jqhuot0r07gdf4f1af1wekmp - Failure occurred on job jqhuxi0q07gdf4f1afcl7uto: Error: Caught error spawning Chromium\n    at Observable.Rx.Observable.create [as _subscribe] (/usr/share/kibana/node_modules/x-pack/plugins/reporting/server/browsers/chromium/driver_factory/index.js:56:24)\n    at <anonymous>\n    at process._tickDomainCallback (internal/process/next_tick.js:228:7)"}
{"type":"log","@timestamp":"2019-01-04T09:48:55Z","tags":["reporting","worker","debug"],"pid":9661,"message":"PDF: Worker error: (jqhuxi0q07gdf4f1afcl7uto)"}
{"type":"log","@timestamp":"2019-01-04T09:48:55Z","tags":["reporting","esqueue","worker","warn"],"pid":9661,"message":"jqhuot0r07gdf4f1af1wekmp - Failing job jqhuxi0q07gdf4f1afcl7uto"}

dao · January 4, 2019, 9:59am

Do you have enough information with:

Error: Caught error spawning Chromium 
at Observable.Rx.Observable.create [as _subscribe] (/usr/share/kibana/node_modules/x-pack/plugins/reporting/server/browsers/chromium/driver_factory/index.js:56:24)
at <anonymous>
at process._tickDomainCallback (internal/process/next_tick.js:228:7)"

dao · January 8, 2019, 8:14am

Hello,

could you find anything interesting with the logs?

dao · January 8, 2019, 10:46am

I have added a log line in the source and I have this. Does it rings a bell (debian 9)?

Failed to launch chrome!

/var/lib/kibana/headless_shell-linux/headless_shell: error while loading shared libraries: libnss3.so: cannot open shared object file: No such file or directory

TROUBLESHOOTING: https://github.com/GoogleChrome/puppeteer/blob/master/docs/troubleshooting.md

joelgriffith · January 8, 2019, 4:57pm

AH, this is perfect! libnss3 is telling sign that you need to install the following package: nss for this to work. Depending on your platform/OS it's as simple as installing: sudo apt-get install -y libnss3.

This command might change depending on your distro (just a heads up), and what package manager it uses.

joelgriffith · January 8, 2019, 6:50pm

@dao what line did you alter to get that message?

dao · January 9, 2019, 7:18am

YEEEESSS! it works now! I think the error message in the logs should be more explicit (the error is not logged).

To find the problem, I have added a few log lines in the code of kibana:

/usr/share/kibana/node_modules/x-pack/plugins/reporting/server/browsers/chromium/driver_factory/index.js

      try {
	const args = JSON.stringify({
		userDataDir, executablePath: this.binaryPath, args: chromiumArgs
	});
	this.logger.info(`We are IN with ${args}`);
        chromium = await puppeteer.launch({
          userDataDir,
          executablePath: this.binaryPath,
          ignoreHTTPSErrors: true,
          args: chromiumArgs,
          env: {
            TZ: browserTimezone
          },
        });
	this.logger.info(`chromium lanchedi IN`);
        page = await chromium.newPage();
	      this.logger.info(`page generated IN`);
      } catch (err) {
        observer.error(new Error(`Caught error spawning Chromium`));
	      this.logger.error("IN");
	      this.logger.error(err);
        return;
      }