Randomly getting Error: Failed to decrypt report job data. with Kibana

javadevmtl · April 30, 2019, 6:42pm

Hi, I have two instances of Kibana 6.4.2 running in Docker container on DC/OS and I randomly get: Error: Failed to decrypt report job data. with Kibana.

Sometimes the reports work, sometimes not.

I have set xpack.reporting.encryptionKey as an environment variable. NOT in the kibana.yml

If it's an environment var should it be: XPACK_REPORTING_ENCRYPTIONKEY?

cjcenizal · April 30, 2019, 11:19pm

If this error is intermittent then you might be running into a problem in which the amount of data used to generate the report is too large. Here's a similar issue in which the user solved the problem by using a smaller time interval. Can you see if this works, to confirm that this is the problem?

javadevmtl · May 1, 2019, 2:50pm

The intermittent also happens on the same report.

The data range is 1 month and produces about 900 documents. On the discover page it seems to take 3 seconds max.

javadevmtl · May 2, 2019, 8:31pm

So, I dunno how much smaller I can do it. Or should I reduce the month to a day type of thing?

joelgriffith · May 6, 2019, 7:15pm

Hey @javadevmtl, do your Kibana instances have the same kibana.index set? There's a known issue (really mis-configuration) where having the different kibana.index's, but the same xpack.reporting.index, will cause Kibana to claim jobs it cannot fulfill.

With different kibana.index settings across Kibana instances connected to the same Elasticsearch cluster, you must also configure a different xpack.reporting.index setting per each unique kibana.index.

Let me know if that helps!

javadevmtl · May 6, 2019, 8:16pm

Hi, no I do not set a different index per Kibana instance and both configs are default.

These are the only ENVIRONMENT var I set...
"NODE_OPTIONS": "xxxxxx",
"ELASTICSEARCH_URL": "xxxxxx",
"XPACK_MONITORING_UI_CONTAINER_ELASTICSEARCH_ENABLED": "false",
"xpack.reporting.encryptionKey": "xxxxxx"

1- So it seems I don't need to do anything?
2- Should xpack.reporting.encryptionKey be in caps and underscores just like the other environment vars?

javadevmtl · May 7, 2019, 4:02pm

@joelgriffith Hi, basically I run Kibana in DC/OS and all I do is scale up or down. So in essence each instance is running the exact same Docker/Marathon/config, I do not inject any special kibana.yaml in the docker container and use mostly default values except for the above mentioned config.

And then access Kibana through Haproxy.

The only thing I'm not sure is that "xpack.reporting.encryptionKey": "xxxxxx" if it's configurable by environment variable and if it should be XPACK_REPORTING_ENCRYPTION_KEY instead?

system · June 4, 2019, 4:02pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

tsullivan · July 8, 2019, 6:51pm

Hi, the answer to your question is that environment variables used for a config setting DO need to be uppercased. The documentation about this can be found in Docker configuration: https://www.elastic.co/guide/en/kibana/current/docker.html#environment-variable-config as environment variables are often used to run customized Docker containers.

If the environment variable with your encryption key can't be found by Kibana, it will generate a random one, which means other Kibana instances have different encryption keys, and they'll only be allowed to decrypt the reports they created... until they restart.

If Kibana is generating a random encryption key, a warning log about this will be printed in the Kibana console at startup.