Generating a CSV file from curl with Kibana, results in just 'Processing' in the output csv file

[eeijlar (John)]

I am using the following shell snipped to download csvs for a a series of queries that I have collected up in Kibana:

                for filename in ${KIBANA_QUERIES}/*.rison; do
                   echo "Processing query file: $filename"
                   CSV_REPORT=$(basename $filename.csv)
                   REPORT_PATH=$(curl -s --insecure -u "elastic:${ELASTIC_SECRET_UMS}" -H "kbn-version: 7.11.2" -XPOST $(cat $filename) | jq -r '.path')
                   echo "Report path: $REPORT_PATH"
                   echo "Downloading report to: $CSV_REPORT"
                   curl -s --insecure -u "elastic:${ELASTIC_SECRET_UMS}" https://${K8S_NODE_IP}:${KIBANA_NODE_PORT}${REPORT_PATH} --output $HOME/Downloads/$CSV_REPORT
                done

When I run this command, the csv report that comes out just contains the work 'Processing'. Is there a way to wait until the csv report is ready? I am version 7.10 of the ELK stack.

[flash1293 (Joe Reuter)]

You can poll the status code of the report path request - once it becomes 200, the report is ready.

[eeijlar (John)]

Hi Joe,

I tried what you suggested like this:

                for filename in ${KIBANA_QUERIES}/*.rison; do
                   echo "Processing query file: $filename"
                   CSV_REPORT=$(basename $filename.csv)
                   REPORT_READY=400
                   while [ "${REPORT_READY}" != "200" ]
                   do
                     REPORT_READY=$(curl -I -s --insecure -u "elastic:${ELASTIC_SECRET_UMS}" -H "kbn-version: 7.11.2" -XPOST $(cat $filename) | grep HTTP | awk '{print $2}')
                     if [ "$REPORT_READY" == "404" ]; then
                         echo "Exiting on 404"
                         break;
                     fi
                     sleep 1
                     echo "Waiting for report to be ready, status $REPORT_READY"
                   done

                   if [ "$REPORT_READY" == 200 ]; then
                      REPORT_PATH=$(curl -s --insecure -u "elastic:${ELASTIC_SECRET_UMS}" -H "kbn-version: 7.11.2" -XPOST $(cat $filename) | jq -r '.path')
                      echo "Report path: $REPORT_PATH"
                      echo "Downloading report to: $CSV_REPORT"
                      curl -s --insecure -u "elastic:${ELASTIC_SECRET_UMS}" https://${K8S_NODE_IP}:${KIBANA_NODE_PORT}${REPORT_PATH} --output $HOME/Downloads/$CSV_REPORT
                   fi
                done

But I get the same result. Perhaps my polling logic is wrong?

[flash1293 (Joe Reuter)]

Hey, sorry for the late answer. In your new script it seems like you aren't extracting the path.

The idea is the following:

• Kick off the report generation with the POST url copied from the UI (send as POST request)
• This returns 200 immediately, and the response contains the path
• Poll the path via GET request, until the response code turns 200
• Once the response code turns 200, the response body will be the final report

Your second script seems to poll the initial post request instead.

My shell scripting game is really low, but something like this:

                for filename in ${KIBANA_QUERIES}/*.rison; do
                   echo "Processing query file: $filename"
                   CSV_REPORT=$(basename $filename.csv)
                   REPORT_PATH=$(curl -s --insecure -u "elastic:${ELASTIC_SECRET_UMS}" -H "kbn-version: 7.11.2" -XPOST $(cat $filename) | jq -r '.path')
                   echo "Report path: $REPORT_PATH"
                   REPORT_READY=400
                   while [ "${REPORT_READY}" != "200" ]
                   do
                     REPORT_READY=$(curl -I -s --insecure -u "elastic:${ELASTIC_SECRET_UMS}" https://${K8S_NODE_IP}:${KIBANA_NODE_PORT}${REPORT_PATH} | grep HTTP | awk '{print $2}')
                     if [ "$REPORT_READY" == "404" ]; then
                         echo "Exiting on 404"
                         break;
                     fi
                     sleep 1
                     echo "Waiting for report to be ready, status $REPORT_READY"
                   done

                   if [ "$REPORT_READY" == 200 ]; then
                      echo "Downloading report to: $CSV_REPORT"
                      curl -s --insecure -u "elastic:${ELASTIC_SECRET_UMS}" https://${K8S_NODE_IP}:${KIBANA_NODE_PORT}${REPORT_PATH} --output $HOME/Downloads/$CSV_REPORT
                   fi
                done

It's downloading the final report twice, it's probably possible to optimize that, but I don't know how (you need to get the status code and the response body in the same call)