Reporting stopped working due to version conflict

[gulycka]

Hi,

recently I encountered issue with reporting plugin from X-Pack.
We are running ELK stack with only Basic license for X-Pack, in version 7.5.2.

Previously, reporting was working without any issues.
Few days ago I noticed that the reports are not processed correctly. When I generate some report, it is not added to Management/Reporting.
I can see some error logs from Kibana showing that there is some error with older report:

{"type":"response","@timestamp":"2020-06-15T09:47:36Z","tags":["api"],"pid":25849,"method":"post","statusCode":200,"req":{"url":"/api/reporting/generate/csv?jobParams=","method":"post","headers":{"host":"127.0.0.1:5601","sec-fetch-dest":"empty","kbn-version":"7.5.2","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36","dnt":"1","content-type":"application/json","accept":"*/*","origin":"https://domain","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","referer":"https://domain/app/kibana","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","x-forwarded-for":"10.10.100.212","x-forwarded-host":"domain","x-forwarded-server":"domain","connection":"Keep-Alive","content-length":"1404","x-ror-kibana-request-path":"/api/reporting/generate/csv","x-ror-kibana-request-method":"post"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"https://domain/app/kibana"},"res":{"statusCode":200,"responseTime":90,"contentLength":9},"message":"POST /api/reporting/generate/csv?jobParams= 200 90ms - 9.0B"}
{"type":"log","@timestamp":"2020-06-15T09:47:40Z","tags":["reporting","esqueue","queue-worker","info"],"pid":25849,"message":"kbgb8bge0jy1d6c375ci88i0 - Job marked as claimed: /.reporting-prod.kibana-2020.06.14/_doc/kbgbdltx0jy1d6c3757zln51"}
{"type":"log","@timestamp":"2020-06-15T09:47:40Z","tags":["reporting","esqueue","queue-worker","info"],"pid":25849,"message":"kbgb8bge0jy1d6c375ci88i0 - Starting job"}
{"type":"log","@timestamp":"2020-06-15T09:47:40Z","tags":["reporting","esqueue","queue-worker","info"],"pid":25849,"message":"kbgb8bge0jy1d6c375ci88i0 - Job execution completed successfully"}
{"type":"log","@timestamp":"2020-06-15T09:47:40Z","tags":["reporting","esqueue","queue-worker","info"],"pid":25849,"message":"kbgb8bge0jy1d6c375ci88i0 - Job data saved successfully: /.reporting-prod.kibana-2020.06.14/_doc/kbgbdltx0jy1d6c3757zln51"}
{"type":"log","@timestamp":"2020-06-15T09:47:46Z","tags":["reporting","esqueue","queue-worker","warning"],"pid":25849,"message":"kbgb8bge0jy1d6c375ci88i0 - _claimPendingJobs encountered a version conflict on updating pending job kbgbdltx0jy1d6c3757zln51: Error: [version_conflict_engine_exception] [kbgbdltx0jy1d6c3757zln51]: version conflict, required seqNo [3], primary term [1]. current document has seqNo [5] and primary term [1], with { index_uuid=\"6t-7lSrBQc2tER5tuZGQ2A\" & shard=\"0\" & index=\".reporting-prod.kibana-2020.06.14\" }\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)\n    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)\n    at IncomingMessage.emit (events.js:194:15)\n    at endReadableNT (_stream_readable.js:1103:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"}

Interesting is the line:

_claimPendingJobs encountered a version conflict on updating pending job kbgbdltx0jy1d6c3757zln51: Error: [version_conflict_engine_exception]

I tried to restart Kibana, to remove and rename Kibana reporting index, to remove .kibana_task_manager indices, but with no luck.
Reporting still failes.

Thanks for any help,
Andrej.

[wylie (Wylie Conlon)]

This error refers to how Elasticsearch does concurrency control: https://www.elastic.co/guide/en/elasticsearch/reference/current/optimistic-concurrency-control.html

I am not an expert on reporting, but it appears to me like there might be something updating the reporting documents and that the job isn't updated correctly. I will see if one of my coworkers can comment further.

[joelgriffith]

Hey there! Reporting does typically use a .reporting-* index, do you happen to have multiple Kibana's running for reporting? Or just a single one? If so, you can try deleting that index, then restarting Kibana (which will re-create the index) and retry your report. Otherwise, all Kibana instances must be using the same reporting index.

[gulycka]

Hi,

previously I used default configuration of reporting index (.reporting.kibana-).
I tried multiple times to stop Kibana service, delete all the reporting (.reporting.kibana-) indices, start Kibana service and try to generate report. The same issue occured again. New reporting index .reporting.kibana-2020.06.07 was created. Probably some old reporting job was stuck at that time...

Then I tried to stop kibana service, delete all reporting indices (.reporting.kibana-) a then change configuration to another reporting index naming (.reporting-prod.kibana-). When I started kibana service, the same issue occured again.

Then I tried to do the same but I also deleted all kibana tasks (index .kibana_task_manager). After that I started kibana service and I generated new report. Previously created reporting index wasn't recreated (2020.06.07), but another index was created (2020.06.14, it was yesterday's index) and the same error logs pops out and report wasn't generated.

I have setup with only single kibana service running behind reverse proxy (apache), with multiple elasticsearch nodes.

This is my kibana reporting configuration:

xpack.reporting.kibanaServer.port: 443
xpack.reporting.kibanaServer.protocol: https
xpack.reporting.kibanaServer.hostname: domain_replaced
xpack.reporting.encryptionKey: "***"
xpack.reporting.index: .reporting-prod

Thanks for any ideas.

[gulycka]

Hi guys,

looks like problem with one of our 3rd party plugins.
After disabling the plugin, everything works just fine.

I guess we can close the issue here.
I'll report it to the plugin maintainer.

Thanks.