Kibana : Not able to generate large reports

Elastic Stack Kibana
1

Hi,

I am using ELK Version 7.17.2 . We are trying to generate reports for more then a million rows CSV.

Getting below error.
Error
Error: Max attempts reached (3). Queue timeout reached.

####Kibana.yml


*server.host: "0.0.0.0"*

*elasticsearch.hosts: ["https://server1:9200/","https://server2:9200/","https:/server3:9200/"]*

*elasticsearch.username: "kibana"*
*elasticsearch.password: "sadsdsdff"*

*server.ssl.enabled: true*
*server.ssl.certificate: /etc/kibana/certs/kibana.crt*
*server.ssl.key: /etc/kibana/certs/kibana.key*

*elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/certs/ca.crt" ]*

*xpack.reporting.enabled: true*
*xpack.reporting.queue.timeout: 3600000*
*xpack.reporting.csv.maxSizeBytes: 1024000000*
*xpack.reporting.csv.scroll.size: 10000*

*server.maxPayloadBytes: 1073741824*
*xpack.reporting.encryptionKey: "sdfghjklxcvbnmvcbnm"*
Heap/Memory Setting for ELastic Search.
`"_nodes":{"total":3,"successful":3,"failed":0},"cluster_name":"omnet-elk-cluster","nodes":{"niTWPGOuTsS7c85EgNRcUQ":{"timestamp":1667509736426,"name":"tlp11670419-es-02","transport_address":"52.162.182.111:9300","host":"52.162.182.111","ip":"52.162.182.111:9300","roles":["data","ingest","master"],"attributes":{"xpack.installed":"true","transform.node":"false"},"jvm":{"timestamp":1667509736426,"uptime_in_millis":42456416,"mem":{"heap_used_in_bytes":8136529920,"heap_used_percent":47,"heap_committed_in_bytes":17179869184,"heap_max_in_bytes":17179869184,"non_heap_used_in_bytes":233397872,"non_heap_committed_in_bytes":239665152,"pools":{"young":{"used_in_bytes":7264534528,"max_in_bytes":0,"peak_used_in_bytes":10276044800,"peak_max_in_bytes":0},"old":{"used_in_bytes":398039040,"max_in_bytes":17179869184,"peak_used_in_bytes":398039040,"peak_max_in_bytes":17179869184},"survivor":{"used_in_bytes":473956352,"max_in_bytes":0,"peak_used_in_bytes":553648128,"peak_max_in_bytes":0}}},"threads":{"count":97,"peak_count":108},"gc":{"collectors":{"young":{"collection_count":155,"collection_time_in_millis":4014},"old":{"collection_count":0,"collection_time_in_millis":0}}},"buffer_pools":{"mapped":{"count":1477,"used_in_bytes":291798662688,"total_capacity_in_bytes":291798662688},"direct":{"count":70,"used_in_bytes":9817569,"total_capacity_in_bytes":9817568},"mapped - 'non-volatile memory'":{"count":0,"used_in_bytes":0,"total_capacity_in_bytes":0}},"classes":{"current_loaded_count":27520,"total_loaded_count":27520,"total_unloaded_count":0}}},"CF5VfM1vQlulroaQJ2Datg":{"timestamp":1667509736429,"name":"tlp11670418-es-01","transport_address":"52.162.182.110:9300","host":"tlp11670418","ip":"52.162.182.110:9300","roles":["data","ingest","master"],"attributes":{"xpack.installed":"true","transform.node":"false"},"jvm":{"timestamp":1667509736429,"uptime_in_millis":43356684,"mem":{"heap_used_in_bytes":4477454224,"heap_used_percent":26,"heap_committed_in_bytes":17179869184,"heap_max_in_bytes":17179869184,"non_heap_used_in_bytes":226178088,"non_heap_committed_in_bytes":232456192,"pools":{"young":{"used_in_bytes":1644167168,"max_in_bytes":0,"peak_used_in_bytes":10292822016,"peak_max_in_bytes":0},"old":{"used_in_bytes":2582545408,"max_in_bytes":17179869184,"peak_used_in_bytes":5149352960,"peak_max_in_bytes":17179869184},"survivor":{"used_in_bytes":250741648,"max_in_bytes":0,"peak_used_in_bytes":257557200,"peak_max_in_bytes":0}}},"threads":{"count":96,"peak_count":109},"gc":{"collectors":{"young":{"collection_count":116,"collection_time_in_millis":1853},"old":{"collection_count":0,"collection_time_in_millis":0}}},"buffer_pools":{"mapped":{"count":1090,"used_in_bytes":197233828667,"total_capacity_in_bytes":197233828667},"direct":{"count":74,"used_in_bytes":9702869,"total_capacity_in_bytes":9702868},"mapped - 'non-volatile memory'":{"count":0,"used_in_bytes":0,"total_capacity_in_bytes":0}},"classes":{"current_loaded_count":26471,"total_loaded_count":26471,"total_unloaded_count":0}}},"A0_2ygZxR3iER8SQOL0zDQ":{"timestamp":1667509736429,"name":"tlp11670420-es-03","transport_address":"52.162.182.112:9300","host":"52.162.182.112","ip":"52.162.182.112:9300","roles":["data","ingest","master"],"attributes":{"xpack.installed":"true","transform.node":"false"},"jvm":{"timestamp":1667509736429,"uptime_in_millis":42285166,"mem":{"heap_used_in_bytes":6894805352,"heap_used_percent":40,"heap_committed_in_bytes":17179869184,"heap_max_in_bytes":17179869184,"non_heap_used_in_bytes":217247472,"non_heap_committed_in_bytes":223019008,"pools":{"young":{"used_in_bytes":2583691264,"max_in_bytes":0,"peak_used_in_bytes":10284433408,"peak_max_in_bytes":0},"old":{"used_in_bytes":4007313408,"max_in_bytes":17179869184,"peak_used_in_bytes":5149438976,"peak_max_in_bytes":17179869184},"survivor":{"used_in_bytes":303800680,"max_in_bytes":0,"peak_used_in_bytes":853255424,"peak_max_in_bytes":0}}},"threads":{"count":92,"peak_count":99},"gc":{"collectors":{"young":{"collection_count":165,"collection_time_in_millis":3966},"old":{"collection_count":0,"collection_time_in_millis":0}}},"buffer_pools":{"mapped":{"count":1168,"used_in_bytes":224610600583,"total_capacity_in_bytes":224610600583},"direct":{"count":75,"used_in_bytes":9821196,"total_capacity_in_bytes":9821195},"mapped - 'non-volatile memory'":{"count":0,"used_in_bytes":0,"total_capacity_in_bytes":0}},"classes":{"current_loaded_count":26196,"total_loaded_count":26196,"total_unloaded_count":0}}}}}`

Kibana logs:-

{"type":"response","@timestamp":"2022-11-03T21:09:48+00:00","tags":[],"pid":5502,"method":"get","statusCode":200,"req":{"url":"/api/reporting/jobs/list?page=0","method":"get","headers":{"host":"62.172.192.114:5601","connection":"keep-alive","sec-ch-ua":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"100\", \"Google Chrome\";v=\"100\"","content-type":"application/json","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","kbn-version":"7.17.2","kbn-system-request":"true","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://62.172.192.114:5601/app/management/insightsAndAlerting/reporting","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"62.7.202.27","userAgent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","referer":"https://62.172.192.114:5601/app/management/insightsAndAlerting/reporting"},"res":{"statusCode":200,"responseTime":1048,"contentLength":7419},"message":"GET /api/reporting/jobs/list?page=0 200 1048ms - 7.2KB"}
{"type":"response","@timestamp":"2022-11-03T21:09:49+00:00","tags":[],"pid":5502,"method":"get","statusCode":302,"req":{"url":"/","method":"get","headers":{"user-agent":"HTTP-Monitor/1.1","connection":"close"},"remoteAddress":"81.144.93.75","userAgent":"HTTP-Monitor/1.1"},"res":{"statusCode":302,"responseTime":2},"message":"GET / 302 2ms"}
{"type":"response","@timestamp":"2022-11-03T21:09:49+00:00","tags":[],"pid":5502,"method":"get","statusCode":200,"req":{"url":"/api/reporting/jobs/count","method":"get","headers":{"host":"62.172.192.114:5601","connection":"keep-alive","sec-ch-ua":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"100\", \"Google Chrome\";v=\"100\"","content-type":"application/json","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","kbn-version":"7.17.2","kbn-system-request":"true","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://62.172.192.114:5601/app/management/insightsAndAlerting/reporting","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"62.7.202.27","userAgent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","referer":"https://62.172.192.114:5601/app/management/insightsAndAlerting/reporting"},"res":{"statusCode":200,"responseTime":17,"contentLength":1},"message":"GET /api/reporting/jobs/count 200 17ms - 1.0B"}
{"type":"response","@timestamp":"2022-11-03T21:09:54+00:00","tags":[],"pid":5502,"method":"get","statusCode":302,"req":{"url":"/","method":"get","headers":{"user-agent":"HTTP-Monitor/1.1","connection":"close"},"remoteAddress":"81.144.93.76","userAgent":"HTTP-Monitor/1.1"},"res":{"statusCode":302,"responseTime":2},"message":"GET / 302 2ms"}
{"type":"response","@timestamp":"2022-11-03T21:09:54+00:00","tags":[],"pid":5502,"method":"get","statusCode":302,"req":{"url":"/","method":"get","headers":{"user-agent":"HTTP-Monitor/1.1","connection":"close"},"remoteAddress":"81.144.93.75","userAgent":"HTTP-Monitor/1.1"},"res":{"statusCode":302,"responseTime":2},"message":"GET / 302 2ms"}
{"type":"response","@timestamp":"2022-11-03T21:09:54+00:00","tags":[],"pid":5502,"method":"get","statusCode":200,"req":{"url":"/api/reporting/jobs/list?page=0","method":"get","headers":{"host":"62.172.192.114:5601","connection":"keep-alive","sec-ch-ua":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"100\", \"Google Chrome\";v=\"100\"","content-type":"application/json","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","kbn-version":"7.17.2","kbn-system-request":"true","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://62.172.192.114:5601/app/management/insightsAndAlerting/reporting","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"62.7.202.27","userAgent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","referer":"https://62.172.192.114:5601/app/management/insightsAndAlerting/reporting"},"res":{"statusCode":200,"responseTime":1006,"contentLength":7419},"message":"GET /api/reporting/jobs/list?page=0 200 1006ms - 7.2KB"}
{"type":"response","@timestamp":"2022-11-03T21:09:56+00:00","tags":[],"pid":5502,"method":"get","statusCode":200,"req":{"url":"/api/reporting/jobs/count","method":"get","headers":{"host":"62.172.192.114:5601","connection":"keep-alive","sec-ch-ua":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"100\", \"Google Chrome\";v=\"100\"","content-type":"application/json","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","kbn-version":"7.17.2","kbn-system-request":"true","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://62.172.192.114:5601/app/management/insightsAndAlerting/reporting","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"62.7.202.27","userAgent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","referer":"https://62.172.192.114:5601/app/management/insightsAndAlerting/reporting"},"res":{"statusCode":200,"responseTime":17,"contentLength":1},"message":"GET /api/reporting/jobs/count 200 17ms - 1.0B"}
{"type":"response","@timestamp":"2022-11-03T21:09:59+00:00","tags":[],"pid":5502,"method":"get","statusCode":302,"req":{"url":"/","method":"get","headers":{"user-agent":"HTTP-Monitor/1.1","connection":"close"},"remoteAddress":"81.144.93.76","userAgent":"HTTP-Monitor/1.1"},"res":{"statusCode":302,"responseTime":2},"message":"GET / 302 2ms"}
{"type":"response","@timestamp":"2022-11-03T21:09:59+00:00","tags":[],"pid":5502,"method":"get","statusCode":302,"req":{"url":"/","method":"get","headers":{"user-agent":"HTTP-Monitor/1.1","connection":"close"},"remoteAddress":"81.144.93.75","userAgent":"HTTP-Monitor/1.1"},"res":{"statusCode":302,"responseTime":1},"message":"GET / 302 1ms"}
{"type":"response","@timestamp":"2022-11-03T21:10:01+00:00","tags":[],"pid":5502,"method":"get","statusCode":200,"req":{"url":"/api/reporting/jobs/list?page=0","method":"get","headers":{"host":"62.172.192.114:5601","connection":"keep-alive","sec-ch-ua":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"100\", \"Google Chrome\";v=\"100\"","content-type":"application/json","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","kbn-version":"7.17.2","kbn-system-request":"true","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://62.172.192.114:5601/app/management/insightsAndAlerting/reporting","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"62.7.202.27","userAgent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","referer":"https://62.172.192.114:5601/app/management/insightsAndAlerting/reporting"},"res":{"statusCode":200,"responseTime":1047,"contentLength":7419},"message":"GET /api/reporting/jobs/list?page=0 200 1047ms - 7.2KB"}
{"type":"response","@timestamp":"2022-11-03T21:10:02+00:00","tags":[],"pid":5502,"method":"get","statusCode":200,"req":{"url":"/api/reporting/jobs/count","method":"get","headers":{"host":"62.172.192.114:5601","connection":"keep-alive","sec-ch-ua":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"100\", \"Google Chrome\";v=\"100\"","content-type":"application/json","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","kbn-version":"7.17.2","kbn-system-request":"true","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://62.172.192.114:5601/app/management/insightsAndAlerting/reporting","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"62.7.202.27","userAgent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","referer":"https://62.172.192.114:5601/app/management/insightsAndAlerting/reporting"},"res":{"statusCode":200,"responseTime":19,"contentLength":1},"message":"GET /api/reporting/jobs/count 200 19ms - 1.0B"}
{"type":"response","@timestamp":"2022-11-03T21:10:04+00:00","tags":[],"pid":5502,"method":"get","statusCode":302,"req":{"url":"/","method":"get","headers":{"user-agent":"HTTP-Monitor/1.1","connection":"close"},"remoteAddress":"81.144.93.76","userAgent":"HTTP-Monitor/1.1"},"res":{"statusCode":302,"responseTime":2},"message":"GET / 302 2ms"}
{"type":"response","@timestamp":"2022-11-03T21:10:04+00:00","tags":[],"pid":5502,"method":"get","statusCode":302,"req":{"url":"/","method":"get","headers":{"user-agent":"HTTP-Monitor/1.1","connection":"close"},"remoteAddress":"81.144.93.75","userAgent":"HTTP-Monitor/1.1"},"res":{"statusCode":302,"responseTime":2},"message":"GET / 302 2ms"}
{"type":"log","@timestamp":"2022-11-03T21:10:05+00:00","tags":["info","plugins","reporting","reports:monitor"],"pid":5502,"message":"la1dueye0g4131e897ddt67d status is [processing] and the expiration time was [2158ms] ago. Rescheduling..."}
{"type":"log","@timestamp":"2022-11-03T21:10:05+00:00","tags":["info","plugins","reporting","reports:monitor"],"pid":5502,"message":"Rescheduling task:la1dueye0g4131e897ddt67d to retry."}
{"type":"log","@timestamp":"2022-11-03T21:10:08+00:00","tags":["error","plugins","reporting","runTask"],"pid":5502,"message":"Error: Max attempts reached (3). Queue timeout reached.\n    at ExecuteReportTask._claimJob (/usr/share/kibana/x-pack/plugins/reporting/server/lib/tasks/execute_report.js:199:19)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n    at Object.run (/usr/share/kibana/x-pack/plugins/reporting/server/lib/tasks/execute_report.js:340:22)\n    at TaskManagerRunner.run (/usr/share/kibana/x-pack/plugins/task_manager/server/task_running/task_runner.js:275:22)"}
{"type":"log","@timestamp":"2022-11-03T21:10:08+00:00","tags":["error","plugins","reporting","runTask"],"pid":5502,"message":"Error: Max attempts reached (3). Queue timeout reached.\n    at ExecuteReportTask._claimJob (/usr/share/kibana/x-pack/plugins/reporting/server/lib/tasks/execute_report.js:199:19)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n    at Object.run (/usr/share/kibana/x-pack/plugins/reporting/server/lib/tasks/execute_report.js:340:22)\n    at TaskManagerRunner.run (/usr/share/kibana/x-pack/plugins/task_manager/server/task_running/task_runner.js:275:22)"}
{"type":"log","@timestamp":"2022-11-03T21:10:08+00:00","tags":["error","plugins","reporting","runTask"],"pid":5502,"message":"Job la1dueye0g4131e897ddt67d could not be claimed. Exiting..."}
{"type":"response","@timestamp":"2022-11-03T21:10:07+00:00","tags":[],"pid":5502,"method":"get","statusCode":200,"req":{"url":"/api/reporting/jobs/list?page=0","method":"get","headers":{"host":"62.172.192.114:5601","connection":"keep-alive","sec-ch-ua":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"100\", \"Google Chrome\";v=\"100\"","content-type":"application/json","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","kbn-version":"7.17.2","kbn-system-request":"true","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://62.172.192.114:5601/app/management/insightsAndAlerting/reporting","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"62.7.202.27","userAgent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","referer":"https://62.172.192.114:5601/app/management/insightsAndAlerting/reporting"},"res":{"statusCode":200,"responseTime":1096,"contentLength":7416},"message":"GET /api/reporting/jobs/list?page=0 200 1096ms - 7.2KB"}
{"type":"response","@timestamp":"2022-11-03T21:10:08+00:00","tags":[],"pid":5502,"method":"get","statusCode":200,"req":{"url":"/api/reporting/jobs/count","method":"get","headers":{"host":"62.172.192.114:5601","connection":"keep-alive","sec-ch-ua":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"100\", \"Google Chrome\";v=\"100\"","content-type":"application/json","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","kbn-version":"7.17.2","kbn-system-request":"true","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://62.172.192.114:5601/app/management/insightsAndAlerting/reporting","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"62.7.202.27","userAgent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","referer":"https://62.172.192.114:5601/app/management/insightsAndAlerting/reporting"},"res":{"statusCode":200,"responseTime":19,"contentLength":1},"message":"GET /api/reporting/jobs/count 200 19ms - 1.0B"}
{"type":"response","@timestamp":"2022-11-03T21:10:09+00:00","tags":[],"pid":5502,"method":"get","statusCode":302,"req":{"url":"/","method":"get","headers":{"user-agent":"HTTP-Monitor/1.1","connection":"close"},"remoteAddress":"81.144.93.76","userAgent":"HTTP-Monitor/1.1"},"res":{"statusCode":302,"responseTime":2},"message":"GET / 302 2ms"}
{"type":"response","@timestamp":"2022-11-03T21:10:09+00:00","tags":[],"pid":5502,"method":"get","statusCode":302,"req":{"url":"/","method":"get","headers":{"user-agent":"HTTP-Monitor/1.1","connection":"close"},"remoteAddress":"81.144.93.75","userAgent":"HTTP-Monitor/1.1"},"res":{"statusCode":302,"responseTime":2},"message":"GET / 302 2ms"}
{"type":"response","@timestamp":"2022-11-03T21:10:14+00:00","tags":[],"pid":5502,"method":"get","statusCode":302,"req":{"url":"/","method":"get","headers":{"user-agent":"HTTP-Monitor/1.1","connection":"close"},"remoteAddress":"81.144.93.76","userAgent":"HTTP-Monitor/1.1"},"res":{"statusCode":302,"responseTime":1},"message":"GET / 302 1ms"}
{"type":"response","@timestamp":"2022-11-03T21:10:13+00:00","tags":[],"pid":5502,"method":"get","statusCode":200,"req":{"url":"/api/reporting/jobs/list?page=0","method":"get","headers":{"host":"62.172.192.114:5601","connection":"keep-alive","sec-ch-ua":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"100\", \"Google Chrome\";v=\"100\"","content-type":"application/json","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","kbn-version":"7.17.2","kbn-system-request":"true","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://62.172.192.114:5601/app/management/insightsAndAlerting/reporting","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"62.7.202.27","userAgent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","referer":"https://62.172.192.114:5601/app/management/insightsAndAlerting/reporting"},"res":{"statusCode":200,"responseTime":1032,"contentLength":7547},"message":"GET /api/reporting/jobs/list?page=0 200 1032ms - 7.4KB"}
{"type":"response","@timestamp":"2022-11-03T21:10:15+00:00","tags":[],"pid":5502,"method":"get","statusCode":302,"req":{"url":"/","method":"get","headers":{"user-agent":"HTTP-Monitor/1.1","connection":"close"},"remoteAddress":"81.144.93.75","userAgent":"HTTP-Monitor/1.1"},"res":{"statusCode":302,"responseTime":2},"message":"GET / 302 2ms"}
{"type":"response","@timestamp":"2022-11-03T21:10:15+00:00","tags":[],"pid":5502,"method":"get","statusCode":200,"req":{"url":"/api/reporting/jobs/count","method":"get","headers":{"host":"62.172.192.114:5601","connection":"keep-alive","sec-ch-ua":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"100\", \"Google Chrome\";v=\"100\"","content-type":"application/json","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","kbn-version":"7.17.2","kbn-system-request":"true","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://62.172.192.114:5601/app/management/insightsAndAlerting/reporting","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"62.7.202.27","userAgent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","referer":"https://62.172.192.114:5601/app/management/insightsAndAlerting/reporting"},"res":{"statusCode":200,"responseTime":18,"contentLength":1},"message":"GET /api/reporting/jobs/count 200 18ms - 1.0B"}
{"type":"response","@timestamp":"2022-11-03T21:10:19+00:00","tags":[],"pid":5502,"method":"get","statusCode":302,"req":{"url":"/","method":"get","headers":{"user-agent":"HTTP-Monitor/1.1","connection":"close"},"remoteAddress":"81.144.93.76","userAgent":"HTTP-Monitor/1.1"},"res":{"statusCode":302,"responseTime":1},"message":"GET / 302 1ms"}
{"type":"response","@timestamp":"2022-11-03T21:10:20+00:00","tags":[],"pid":5502,"method":"get","statusCode":302,"req":{"url":"/","method":"get","headers":{"user-agent":"HTTP-Monitor/1.1","connection":"close"},"remoteAddress":"81.144.93.75","userAgent":"HTTP-Monitor/1.1"},"res":{"statusCode":302,"responseTime":2},"message":"GET / 302 2ms"}
{"type":"response","@timestamp":"2022-11-03T21:10:20+00:00","tags":[],"pid":5502,"method":"get","statusCode":200,"req":{"url":"/api/reporting/jobs/info/la1dueye0g4131e897ddt67d","method":"get","headers":{"host":"62.172.192.114:5601","connection":"keep-alive","sec-ch-ua":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"100\", \"Google Chrome\";v=\"100\"","content-type":"application/json","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","kbn-version":"7.17.2","kbn-system-request":"true","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://62.172.192.114:5601/app/management/insightsAndAlerting/reporting","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"62.7.202.27","userAgent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","referer":"https://62.172.192.114:5601/app/management/insightsAndAlerting/reporting"},"res":{"statusCode":200,"responseTime":20,"contentLength":1454},"message":"GET /api/reporting/jobs/info/la1dueye0g4131e897ddt67d 200 20ms - 1.4KB"}
{"type":"response","@timestamp":"2022-11-03T21:10:20+00:00","tags":[],"pid":5502,"method":"get","statusCode":200,"req":{"url":"/api/reporting/jobs/list?page=0","method":"get","headers":{"host":"62.172.192.114:5601","connection":"keep-alive","sec-ch-ua":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"100\", \"Google Chrome\";v=\"100\"","content-type":"application/json","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","kbn-version":"7.17.2","kbn-system-request":"true","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://62.172.192.114:5601/app/management/insightsAndAlerting/reporting","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"62.7.202.27","userAgent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","referer":"https://62.172.192.114:5601/app/management/insightsAndAlerting/reporting"},"res":{"statusCode":200,"responseTime":1082,"contentLength":7547},"message":"GET /api/reporting/jobs/list?page=0 200 1082ms - 7.4KB"}
{"type":"response","@timestamp":"2022-11-03T21:10:21+00:00","tags":[],"pid":5502,"method":"get","statusCode":200,"req":{"url":"/api/reporting/jobs/count","method":"get","headers":{"host":"62.172.192.114:5601","connection":"keep-alive","sec-ch-ua":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"100\", \"Google Chrome\";v=\"100\"","content-type":"application/json","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","kbn-version":"7.17.2","kbn-system-request":"true","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://62.172.192.114:5601/app/management/insightsAndAlerting/reporting","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"62.7.202.27","userAgent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","referer":"https://62.172.192.114:5601/app/management/insightsAndAlerting/reporting"},"res":{"statusCode":200,"responseTime":17,"contentLength":1},"message":"GET /api/reporting/jobs/count 200 17ms - 1.0B"}
{"type":"response","@timestamp":"2022-11-03T21:10:24+00:00","tags":[],"pid":5502,"method":"get","statusCode":302,"req":{"url":"/","method":"get","headers":{"user-agent":"HTTP-Monitor/1.1","connection":"close"},"remoteAddress":"81.144.93.76","userAgent":"HTTP-Monitor/1.1"},"res":{"statusCode":302,"responseTime":2},"message":"GET / 302 2ms"}
{"type":"response","@timestamp":"2022-11-03T21:10:25+00:00","tags":[],"pid":5502,"method":"get","statusCode":302,"req":{"url":"/","method":"get","headers":{"user-agent":"HTTP-Monitor/1.1","connection":"close"},"remoteAddress":"81.144.93.75","userAgent":"HTTP-Monitor/1.1"},"res":{"statusCode":302,"responseTime":2},"message":"GET / 302 2ms"}
{"type":"response","@timestamp":"2022-11-03T21:10:26+00:00","tags":[],"pid":5502,"method":"get","statusCode":200,"req":{"url":"/api/reporting/jobs/list?page=0","method":"get","headers":{"host":"62.172.192.114:5601","connection":"keep-alive","sec-ch-ua":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"100\", \"Google Chrome\";v=\"100\"","content-type":"application/json","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","kbn-version":"7.17.2","kbn-system-request":"true","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://62.172.192.114:5601/app/management/insightsAndAlerting/reporting","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"62.7.202.27","userAgent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","referer":"https://62.172.192.114:5601/app/management/insightsAndAlerting/reporting"},"res":{"statusCode":200,"responseTime":1033,"contentLength":7547},"message":"GET /api/reporting/jobs/list?page=0 200 1033ms - 7.4KB"}
{"type":"response","@timestamp":"2022-11-03T21:10:27+00:00","tags":[],"pid":5502,"method":"get","statusCode":200,"req":{"url":"/api/reporting/jobs/count","method":"get","headers":{"host":"62.172.192.114:5601","connection":"keep-alive","sec-ch-ua":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"100\", \"Google Chrome\";v=\"100\"","content-type":"application/json","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","kbn-version":"7.17.2","kbn-system-request":"true","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://62.172.192.114:5601/app/management/insightsAndAlerting/reporting","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"62.7.202.27","userAgent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","referer":"https://62.172.192.114:5601/app/management/insightsAndAlerting/reporting"},"res":{"statusCode":200,"responseTime":19,"contentLength":1},"message":"GET /api/reporting/jobs/count 200 19ms - 1.0B"}
{"type":"response","@timestamp":"2022-11-03T21:10:29+00:00","tags":[],"pid":5502,"method":"get","statusCode":302,"req":{"url":"/","method":"get","headers":{"user-agent":"HTTP-Monitor/1.1","connection":"close"},"remoteAddress":"81.144.93.76","userAgent":"HTTP-Monitor/1.1"},"res":{"statusCode":302,"responseTime":2},"message":"GET / 302 2ms"}
2

Hi @shobhit !

That is quite a lot of data. It looks like the error message is complaining about hitting your queue timeout of 60 minutes. FWIW, you should also be able to provide a string like 60m for this piece of config.

How long is the report running for? Increasing xpack.reporting.queue.timeout might help, but there might be other issues with such a large export.

3

Hello (@JLeysens),

Thanks for your response.

Report runs for 1 hour and the fails , it tries for 3 times. I have increased the Heap sizes on Elastic to 16 GB. Do i need to change any thing for Kibana ?

In kibana.yml

## x-pack module setting

xpack.reporting.queue.timeout: 3600000

xpack.reporting.csv.maxSizeBytes: 1024000000

xpack.reporting.csv.scroll.size: 10000

server.maxPayloadBytes: 1073741824

and this is the configuration in elasticsearch.yml
http.compression: true
http.max_content_length: 1024mb

Title
Untitled discover search
Created at
2022-11-03 @ 06:09 PM
elastic
Status
Failed at 2022-11-03 @ 09:10 PM
Time zone
Europe/London
Started at
2022-11-03T20:10:03.211Z
Completed at
2022-11-03T21:10:08.387Z
Processed by
tlpxxxxxx (4cc52c3d-8356-42b0-8e45-d23cd3d34e65)
Content type
n/a
Size in bytes
n/a
Attempts
3
Max attempts
3
Timeout
3600000
Export type
csv_searchsource
Error
Error: Max attempts reached (3). Queue timeout reached.

4

Hello @Agi_K_Thomas , @JLeysens , @stephenb

Could you please help on this one.

Regards,
Shobhit

5

Hi @shobhit please don't invite people to directly join that have not joined on their own

Personally I would not try to use Kibana to download 1M plus documents... It's not really designed for that.. might you get it to work with a lot of tweaking maybe... Not my area of expertise.

I would perhaps use logstash with elasticsearch input and csv output.. probably would take a couple minutes.

This is really more of an export use case not reporting.

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.