We recently upgraded our Elastic stack from 6.8 to 7.15 and one thing we've noticed is a pretty noticeable degradation in the performance of CSV reporting in Kibana; reports we were usually generating successfully in a few minutes without impact we're now seeing take a very long time (or never completing), and consuming enough CPU that the Kibana instance running the report becomes unresponsive.
The example below is a report with ~400k rows which I appreciate is a lot but we had been able to make CSV reports with this many rows prior to the upgrade. As you can see, CPU usage grows massively while the report is running until eventually our liveness probe kills the Kibana container as it is unresponsive
Unfortunately, I don't have any historic metrics to share from when we were on 6.8 to compare with.
I've seen you can have reporting only nodes however we don't use reports frequent enough to warrant having dedicated nodes for them.
As you can see the report was running for ~40 minutes before dying, historically this report would have completed much sooner than that
If there's any particular debug information that could be useful to help diagnose please let me know
Kibana version: v 7.15.1
Additional reporting config:
- name: XPACK_REPORTING_ENCRYPTIONKEY value: <redacted> - name: XPACK_REPORTING_QUEUE_TIMEOUT value: "3600000" - name: XPACK_REPORTING_CSV_MAXSIZEBYTES value: "524288000"