Kibana Report Generation Timeout

[TASI (Thomas Pali)]

We currently have Elasticsearch, Logstash and Kibana installed (version 7.16.1).. We use scripts to generate and download data from Kibana/Elasticsearch using curl and a POST URL.

One intermittent issue we have been facing is that sometimes one of our reports comes up with no data and the contents of the downloaded file just says "processing". I am wondering if there is a timeout setting within Kibana or some other mechanism that could allow more time for the report to run?

This only seems to happen on days where there is a lot of data in the reports.

Any help would be greatly appreciated.

[tsullivan (Tim Sullivan)]

@TASI

our reports comes up with no data and the contents of the downloaded file just says "processing"

It sounds like Kibana is still processing the report. There should be a 503 as the response code when this is the case. Your scripts need to check the HTTP response codes to coordinate with Kibana and download the report when it is done.

See Automatically generate reports | Kibana Guide [8.11] | Elastic

[TASI (Thomas Pali)]

Thanks for the response. Right now my script does not check the HTTP response code prior to attempting the download. We just generate the Kibana post url using curl -X POST, have our script wait a predefined amount of time, then attempt to download the report via another curl. We tried adjusting the wait time but no matter how long we set it, it never seems to finish on time.

Would you recommend restructuring our script to take the HTTP response code into consideration? Personally, I have never done this so I would need to research how this is done.

[tsullivan (Tim Sullivan)]

Heya, I'm not so comfortable reviewing your scripts. Instead, I'd like to just share a shell script which uses curl and jq to do the job:

#!/bin/bash
#set -x

HOST=https://example.com
AUTH=test_user:changeme
PREFIX=/kibana

POST_URL="${HOST}${PREFIX}/api/reporting/generate/csv_searchsource?jobParams=....."

DOWNLOAD_PATH=${HOST}$(curl --silent -XPOST "$POST_URL" -H "kbn-xsrf: get_report" -u "$AUTH" | jq -r ".path")

echo Download path: $DOWNLOAD_PATH

STATUS=''
while [[ -z $STATUS || $STATUS =~ .*503.* ]]
do
  STATUS=$(curl --silent --head "$DOWNLOAD_PATH" -u "$AUTH" | head -1)
  echo $STATUS
done

# download final report
curl "$DOWNLOAD_PATH" -u "$AUTH"