Kibana Reporting Max Attempts reached (3) error

Elastic Stack Kibana
1

I am using bitnami ELK stack and have installed x-pack both on elasticsearch and kibana

When I am generating reports from a visualization it gives me the following error

image
image.png949×93 6.98 KB

I have following properties in kibana.yml

xpack.reporting.queue.timeout: 600000

any help will be highly appreciated. I do not see anything in elasticsearch logs.

There is no message related to pdf after the following log statement

{"type":"log","@timestamp":"2018-03-19T04:12:32Z","tags":["reporting","worker","debug"],"pid":74889,"message":"PDF: Processing PDF job"}

2

Hey,

could you maybe paste a bit more of the log context around that. Especially everything that comes after it. It doesn't necessarily need to have the word PDF in it, even if its relevant for reporting.

Cheers,
Tim

3

hey Tim,

here is the log

{"type":"log","@timestamp":"2018-03-20T01:26:25Z","tags":["reporting","worker","debug"],"pid":91601,"message":"PDF: Processing PDF job"}
{"type":"log","@timestamp":"2018-03-20T01:26:26Z","tags":["plugin","debug"],"pid":91601,"message":"Checking Elasticsearch version"}
{"type":"ops","@timestamp":"2018-03-20T01:26:26Z","tags":,"pid":91601,"os":{"load":[0.193359375,0.1728515625,0.15380859375],"mem":{"total":3585667072,"free":139182080},"uptime":341276},"proc":{"uptime":97.664,"mem":{"rss":226922496,"heapTotal":180592640,"heapUsed":146399864,"external":2373687},"delay":0.48129498958587646},"load":{"requests":{"5601":{"total":2,"disconnects":0,"statusCodes":{"200":2}}},"concurrents":{"5601":15},"responseTimes":{"5601":{"avg":126,"max":238}},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 139.6MB uptime: 0:01:38 load: [0.19 0.17 0.15] delay: 0.481"}
{"type":"log","@timestamp":"2018-03-20T01:26:26Z","tags":["debug","monitoring-ui","kibana-monitoring"],"pid":91601,"message":"Received Kibana Ops event data"}
{"type":"response","@timestamp":"2018-03-20T01:26:27Z","tags":,"pid":91601,"method":"get","statusCode":200,"req":{"url":"/api/reporting/jobs/list?page=0","method":"get","headers":{"host":"127.0.0.1:5601","accept":"application/json, text/plain, /","kbn-system-api":"true","kbn-version":"6.1.2","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36","referer":"http://{IPAddress}/elk/app/kibana","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9","x-forwarded-for":"142.116.220.222","x-forwarded-host":"{IPAddress}","x-forwarded-server":"localhost","connection":"Keep-Alive"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://{IPAddress}/elk/app/kibana"},"res":{"statusCode":200,"responseTime":39,"contentLength":9},"message":"GET /api/reporting/jobs/list?page=0 200 39ms - 9.0B"}
{"type":"log","@timestamp":"2018-03-20T01:26:27Z","tags":["reporting","screenshots","debug"],"pid":91601,"message":"opening http://127.0.0.1:5601/elk/app/kibana#/visualize/edit/7ae08de0-2351-11e8-8e57-5ffd065fc8ea?_g=(time:(from:'2018-03-19T00:00:00.000Z',mode:absolute,to:'2018-03-19T23:59:59.999Z'))&_a=(filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'4abf4d70-22e0-11e8-8e57-5ffd065fc8ea',key:fields.RequestUrl.keyword,negate:!f,params:!('http:%2F%2Flocalhost:3000%2Fauthenticate','http:%2F%2F127.0.0.1:3000%2Fauthenticate'),type:phrases,value:'http:%2F%2Flocalhost:3000%2Fauthenticate,%20http:%2F%2F127.0.0.1:3000%2Fauthenticate'),query:(bool:(minimum_should_match:1,should:!((match_phrase:(fields.RequestUrl.keyword:'http:%2F%2Flocalhost:3000%2Fauthenticate')),(match_phrase:(fields.RequestUrl.keyword:'http:%2F%2F127.0.0.1:3000%2Fauthenticate')))))),('$state':(store:appState),exists:(field:fields.AccountID.keyword),meta:(alias:!n,disabled:!f,index:'4abf4d70-22e0-11e8-8e57-5ffd065fc8ea',key:fields.AccountID.keyword,negate:!f,type:exists,value:exists))),linked:!f,query:(language:lucene,query:''),uiState:(),vis:(aggs:!((enabled:!t,id:'1',params:(),schema:metric,type:count),(enabled:!t,id:'2',params:(customInterval:'2h',extended_bounds:(),field:'@timestamp',interval:auto,min_doc_count:1),schema:segment,type:date_histogram),(enabled:!t,id:'3',params:(field:fields.RequestBody.uuid.keyword,order:desc,orderBy:'1',row:!t,size:5),schema:split,type:terms)),params:(addLegend:!t,addTimeMarker:!f,addTooltip:!t,categoryAxes:!((id:CategoryAxis-1,labels:(show:!t,truncate:100),position:bottom,scale:(type:linear),show:!t,style:(),title:(),type:category)),grid:(categoryLines:!f,style:(color:%23eee)),legendPosition:right,radiusRatio:51,seriesParams:!((data:(id:'1',label:Count),drawLinesBetweenPoints:!t,mode:stacked,show:true,showCircles:!t,type:histogram,valueAxis:ValueAxis-1)),times:!(),type:histogram,valueAxes:!((id:ValueAxis-1,labels:(filter:!f,rotate:0,show:!t,truncate:100),name:LeftAxis-1,position:left,scale:(mode:normal,type:linear),show:!t,style:(),title:(text:Count),type:value))),title:'Number%20of%20logins%20(including%20repeated%20logins)',type:histogram))"}
{"type":"log","@timestamp":"2018-03-20T01:26:27Z","tags":["license","debug","xpack"],"pid":91601,"message":"Calling Elasticsearch _xpack API"}
{"type":"response","@timestamp":"2018-03-20T01:26:27Z","tags":,"pid":91601,"method":"get","statusCode":404,"req":{"url":"/elk/app/kibana","method":"get","headers":{"accept":"application/json, text/plain, /","accept-language":"en-US,en;q=0.9","connection":"Keep-Alive","kbn-version":"6.1.2","origin":"http://52.169.110.171","referer":"http://52.169.110.171/elk/app/kibana","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36","x-forwarded-for":"142.116.220.222","x-forwarded-host":"52.169.110.171","x-forwarded-server":"localhost","accept-encoding":"gzip, deflate","host":"127.0.0.1:5601"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://52.169.110.171/elk/app/kibana"},"res":{"statusCode":404,"responseTime":13,"contentLength":9},"message":"**GET /elk/app/kibana 404 13ms** - 9.0B"}

{"type":"log","@timestamp":"2018-03-20T01:28:32Z","tags":["reporting","worker","debug"],"pid":91601,"message":"PDF: Processing PDF job"}
{"type":"log","@timestamp":"2018-03-20T01:28:32Z","tags":["reporting","esqueue","worker","debug"],"pid":91601,"message":"jeyz8iq21yoh23957f86dyqd - querying for outstanding jobs"}
{"type":"log","@timestamp":"2018-03-20T01:28:32Z","tags":["reporting","esqueue","worker","debug"],"pid":91601,"message":"jeyz8iq21yoh23957f86dyqd - 0 outstanding jobs returned"}
{"type":"response","@timestamp":"2018-03-20T01:28:32Z","tags":,"pid":91601,"method":"get","statusCode":200,"req":{"url":"/api/reporting/jobs/list?page=0","method":"get","headers":{"host":"127.0.0.1:5601","accept":"application/json, text/plain, /","kbn-system-api":"true","kbn-version":"6.1.2","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36","referer":"http://{IPAddress}/elk/app/kibana","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9","x-forwarded-for":"142.116.220.222","x-forwarded-host":"{IPAddress}","x-forwarded-server":"localhost","connection":"Keep-Alive"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://{IPAddress}/elk/app/kibana"},"res":{"statusCode":200,"responseTime":16,"contentLength":9},"message":"GET /api/reporting/jobs/list?page=0 200 16ms - 9.0B"}

4

Hi,

after syncing with some colleagues I think the actual issue is, that you are running behind a reverse proxy. If you are running behind a reverse proxy (as you do), you need to provide some addition information in the kibana.yml configuration for reporting - see docs. The actual values might depend on your actual architecture. In that case you should consult the bitnami docs or forums, on how those values must be set.

Cheers,
Tim

5

In case someone needs help..here is the solution

1 Like
6

Thanks for sharing that post. That will be very helpful to guide other Bitnami users! :elasticheart:

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.