Kibana Reports Hammering CPU

[Evesy (Michael Eves)]

Hi,

We recently upgraded our Elastic stack from 6.8 to 7.15 and one thing we've noticed is a pretty noticeable degradation in the performance of CSV reporting in Kibana; reports we were usually generating successfully in a few minutes without impact we're now seeing take a very long time (or never completing), and consuming enough CPU that the Kibana instance running the report becomes unresponsive.

The example below is a report with ~400k rows which I appreciate is a lot but we had been able to make CSV reports with this many rows prior to the upgrade. As you can see, CPU usage grows massively while the report is running until eventually our liveness probe kills the Kibana container as it is unresponsive

Unfortunately, I don't have any historic metrics to share from when we were on 6.8 to compare with.
I've seen you can have reporting only nodes however we don't use reports frequent enough to warrant having dedicated nodes for them.

As you can see the report was running for ~40 minutes before dying, historically this report would have completed much sooner than that

If there's any particular debug information that could be useful to help diagnose please let me know

Kibana version: v 7.15.1

Additional reporting config:

    - name: XPACK_REPORTING_ENCRYPTIONKEY
      value: <redacted>
    - name: XPACK_REPORTING_QUEUE_TIMEOUT
      value: "3600000"
    - name: XPACK_REPORTING_CSV_MAXSIZEBYTES
      value: "524288000"

Cheers

[cheiligers]

HI @Evesy It looks like you've run into a similar issue that's being discussed. I recommend adding your findings to that issue to help the team investigate. Any and all input will help us out!

Do you see any errors/warnings in your server logs? You can enable logging for reporting by adding this to your kibana.yml:

logging:
  appenders:
    console:
      type: console
      layout:
        type: json
  root:
    appenders: [console]
    level: info
 loggers:
    - name: plugins.reporting
      appenders: [console]
      level: debug

That logging config will give the most info but it's going to be very verbose! You might want to output the logs to file using the file appender.

[Evesy (Michael Eves)]

Thanks @cheiligers

That does sound similar, although in our case we're finding we have to delete the report before it manages to complete due to the impact it has on users performing other Kibana activities, so I couldn't say for sure if the report would be truncated or not.

I will add some additional loggers and report back on the issue anything I find of use