POST URL reporting resulting in 'processing'

Elastic Stack Kibana
1

I have been using a self-made script that uses a generated POST URL from Kibana to create and email daily reports from a winlogbeat index. It has been working fine for months but suddenly the report only has 'processing' when opened.

I thought one fix was to extend the sleep command in the script from 3m to 5m which would give more time for the POST URL data to download before doing the curl. This initially fixed our 'processing' issue but it was soon back to 'processing'. Also I want to mention I increased the sleep timer past 5m but was presented with this error:

{"statusCode":500,"error":"Internal Server Error","message":"Reporting generation failed: Max attempts reached (3)"}

Can anyone assist with this issue?

2

Welcome to our community! :smiley:

What version are you on?
What do the Kibana logs show?

3

The version I am running is 7.13.0 and the Kibana journalctl has no entries even after running the reporting script.

Edit: Actually in my /var/log/kibana I have logs related to the script failure. Here they are:

{"type":"response","@timestamp":"2021-09-29T09:23:04-10:00","tags":[],"pid":1574,"method":"get","statusCode":503,"req":{"url":"/api/reporting/jobs/download/ku5wbunh017q8558112pm9lv","method":"get","headers":{"user-agent":"curl/7.29.0","host":"10.100.10.36:5601","accept":"*/*"},"remoteAddress":"10.100.10.36","userAgent":"curl/7.29.0"},"res":{"statusCode":503,"responseTime":821,"contentLength":7},"message":"GET /api/reporting/jobs/download/ku5wbunh017q8558112pm9lv 503 821ms - 7.0B"}
{"type":"log","@timestamp":"2021-09-29T09:24:28-10:00","tags":["info","plugins","reporting","queue-job"],"pid":1574,"message":"Queued csv_searchsource report: ku5wdo6y017q8558111q6cwv"}
{"type":"response","@timestamp":"2021-09-29T09:24:28-10:00","tags":[],"pid":1574,"method":"post","statusCode":200,"req":{"url":"/api/reporting/generate/csv_searchsource?jobParams=%28browserTimezone%3APacific%2FHonolulu%2Ccolumns%3A%21%28message%2Cuser.name%2Chost.name%2C%27%40timestamp%27%29%2CobjectType%3Asearch%2CsearchSource%3A%28fields%3A%21%28%28field%3A%27%2A%27%2Cinclude_unmapped%3Atrue%29%29%2Cfilter%3A%21%28%28meta%3A%28index%3A%27winlogbeat-%2A%27%2Cparams%3A%28%29%29%2Crange%3A%28%27%40timestamp%27%3A%28format%3Astrict_date_optional_time%2Cgte%3Anow-1d%2Clte%3Anow%29%29%29%29%2Cindex%3A%27winlogbeat-%2A%27%2Cparent%3A%28filter%3A%21%28%29%2ChighlightAll%3A%21t%2Cindex%3A%27winlogbeat-%2A%27%2Cquery%3A%28language%3Akuery%2Cquery%3A%27NOT%20%22netmon%22%20AND%20NOT%20%22Administrator%22%20AND%20NOT%20%22jumpsrv%22%20AND%20NOT%20%22jcuser%22%20AND%20NOT%20%2A%24%20AND%20NOT%20%22ldapuser%22%20AND%20NOT%20%22service-sqladmin%22%20AND%20NOT%20%22relay%22%20AND%20NOT%20%22relay2%22%20AND%20NOT%20%22ANONYMOUS%20LOGON%22%20AND%20%28%22An%20account%20was%20logged%20off.%22%20OR%20%22A%20logon%20was%20attempted%20using%20explicit%20credentials.%22%20OR%20%22An%20account%20was%20successfully%20logged%20on.%22%29%27%29%2Cversion%3A%21t%29%2Csort%3A%21%28%28%27%40timestamp%27%3Adesc%29%29%2Cversion%3A%21t%29%2Ctitle%3A%27DCON%20logins%20%2B%20logouts%20%2B%20denials%27%29","method":"post","headers":{"user-agent":"curl/7.29.0","host":"10.100.10.36:5601","accept":"*/*","kbn-version":"7.13.0"},"remoteAddress":"10.100.10.36","userAgent":"curl/7.29.0"},"res":{"statusCode":200,"responseTime":533,"contentLength":1593},"message":"POST /api/reporting/generate/csv_searchsource?jobParams=%28browserTimezone%3APacific%2FHonolulu%2Ccolumns%3A%21%28message%2Cuser.name%2Chost.name%2C%27%40timestamp%27%29%2CobjectType%3Asearch%2CsearchSource%3A%28fields%3A%21%28%28field%3A%27%2A%27%2Cinclude_unmapped%3Atrue%29%29%2Cfilter%3A%21%28%28meta%3A%28index%3A%27winlogbeat-%2A%27%2Cparams%3A%28%29%29%2Crange%3A%28%27%40timestamp%27%3A%28format%3Astrict_date_optional_time%2Cgte%3Anow-1d%2Clte%3Anow%29%29%29%29%2Cindex%3A%27winlogbeat-%2A%27%2Cparent%3A%28filter%3A%21%28%29%2ChighlightAll%3A%21t%2Cindex%3A%27winlogbeat-%2A%27%2Cquery%3A%28language%3Akuery%2Cquery%3A%27NOT%20%22netmon%22%20AND%20NOT%20%22Administrator%22%20AND%20NOT%20%22jumpsrv%22%20AND%20NOT%20%22jcuser%22%20AND%20NOT%20%2A%24%20AND%20NOT%20%22ldapuser%22%20AND%20NOT%20%22service-sqladmin%22%20AND%20NOT%20%22relay%22%20AND%20NOT%20%22relay2%22%20AND%20NOT%20%22ANONYMOUS%20LOGON%22%20AND%20%28%22An%20account%20was%20logged%20off.%22%20OR%20%22A%20logon%20was%20attempted%20using%20explicit%20credentials.%22%20OR%20%22An%20account%20was%20successfully%20logged%20on.%22%29%27%29%2Cversion%3A%21t%29%2Csort%3A%21%28%28%27%40timestamp%27%3Adesc%29%29%2Cversion%3A%21t%29%2Ctitle%3A%27DCON%20logins%20%2B%20logouts%20%2B%20denials%27%29 200 533ms - 1.6KB"}
{"type":"response","@timestamp":"2021-09-29T09:24:29-10:00","tags":[],"pid":1574,"method":"get","statusCode":503,"req":{"url":"/api/reporting/jobs/download/ku5wdo6y017q8558111q6cwv","method":"get","headers":{"user-agent":"curl/7.29.0","host":"10.100.10.36:5601","accept":"*/*"},"remoteAddress":"10.100.10.36","userAgent":"curl/7.29.0"},"res":{"statusCode":503,"responseTime":248,"contentLength":7},"message":"GET /api/reporting/jobs/download/ku5wdo6y017q8558111q6cwv 503 248ms - 7.0B"}
{"type":"log","@timestamp":"2021-09-29T09:25:04-10:00","tags":["warning","plugins","reporting","esqueue","queue-worker"],"pid":1574,"message":"ktui9p04017q855811g2gu5z - Timeout processing job ku5wbunh017q8558112pm9lv"}
{"type":"log","@timestamp":"2021-09-29T09:25:04-10:00","tags":["warning","plugins","reporting","esqueue","queue-worker"],"pid":1574,"message":"ktui9p04017q855811g2gu5z - Timeout on job ku5wbunh017q8558112pm9lv"}
{"type":"log","@timestamp":"2021-09-29T09:25:09-10:00","tags":["info","plugins","reporting","esqueue","queue-worker"],"pid":1574,"message":"ktui9p04017q855811g2gu5z - Job execution completed successfully"}
{"type":"log","@timestamp":"2021-09-29T09:25:10-10:00","tags":["info","plugins","reporting","esqueue","queue-worker"],"pid":1574,"message":"ktui9p04017q855811g2gu5z - Job marked as claimed: /.reporting-2021-09-26/_doc/ku5wbunh017q8558112pm9lv"}
{"type":"log","@timestamp":"2021-09-29T09:25:10-10:00","tags":["info","plugins","reporting","esqueue","queue-worker"],"pid":1574,"message":"ktui9p04017q855811g2gu5z - Starting job"}
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.