500 Internal Server Error while using Reporting

Elastic Stack Kibana
1

I am running a 3 node cluster with 1 kibana instance on node 1. While node 2 is master. Monitoring works and my whole clusterhealth is green

When generating a report in Kibana I get this error message:
Reporting: Error 500 : An internal server error occurred

Kibana logs shows the following error:

{
            	"type": "error",
            	"@timestamp": "2018-01-26T13:57:38Z",
            	"tags": [],
            	"pid": 1027,
            	"level": "error",
            	"error": {
            		"message": "[remote_transport_exception] [node-2][IPNODE2:9300][indices:admin/create]",
            		"name": "Error",
            		"stack": "[remote_transport_exception] [node-2][IPNODE2:9300][indices:admin/create] :: {\"path\":\"/.reporting-2018.01.21\",\"query\":{},\"body\":\"{\\\"settings\\\":{},\\\"mappings\\\":{\\\"esqueue\\\":{\\\"properties\\\":{\\\"jobtype\\\":{\\\"type\\\":\\\"keyword\\\"},\\\"payload\\\":{\\\"type\\\":\\\"object\\\",\\\"enabled\\\":false},\\\"priority\\\":{\\\"type\\\":\\\"byte\\\"},\\\"timeout\\\":{\\\"type\\\":\\\"long\\\"},\\\"process_expiration\\\":{\\\"type\\\":\\\"date\\\"},\\\"created_by\\\":{\\\"type\\\":\\\"keyword\\\"},\\\"created_at\\\":{\\\"type\\\":\\\"date\\\"},\\\"started_at\\\":{\\\"type\\\":\\\"date\\\"},\\\"completed_at\\\":{\\\"type\\\":\\\"date\\\"},\\\"attempts\\\":{\\\"type\\\":\\\"short\\\"},\\\"max_attempts\\\":{\\\"type\\\":\\\"short\\\"},\\\"status\\\":{\\\"type\\\":\\\"keyword\\\"},\\\"output\\\":{\\\"type\\\":\\\"object\\\",\\\"properties\\\":{\\\"content_type\\\":{\\\"type\\\":\\\"keyword\\\"},\\\"content\\\":{\\\"type\\\":\\\"object\\\",\\\"enabled\\\":false}}}}}}}\",\"statusCode\":400,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"remote_transport_exception\\\",\\\"reason\\\":\\\"[node-2][IPNODE2:9300][indices:admin/create]\\\"}],\\\"type\\\":\\\"illegal_argument_exception\\\",\\\"reason\\\":\\\"Rejecting mapping update to [.reporting-2018.01.21] as the final mapping would have more than 1 type: [esqueue, doc]\\\"},\\\"status\\\":400}\"}\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:295:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:254:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:159:7)\n    at IncomingMessage.bound (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js:729:21)\n    at emitNone (events.js:91:20)\n    at IncomingMessage.emit (events.js:185:7)\n    at endReadableNT (_stream_readable.js:974:12)\n    at _combinedTickCallback (internal/process/next_tick.js:80:11)\n    at process._tickDomainCallback (internal/process/next_tick.js:128:9)"
            	},
            	"url": {
            		"protocol": null,
            		"slashes": null,
            		"auth": null,
            		"host": null,
            		"port": null,
            		"hostname": null,
            		"hash": null,
            		"search": "?jobParams=(conflictedTypesFields:!(),fields:!(ip,variable2),indexPatternId:d58f2a10-ff77-11e7-ae73-fd89243a1cfc,metaFields:!(_source,_id,_type,_index,_score),searchRequest:(body:(_source:(excludes:!(),includes:!(ip,variable2)),docvalue_fields:!(),query:(bool:(filter:!(),must:!((match_all:())),must_not:!(),should:!())),script_fields:(),sort:!((_score:(order:desc))),stored_fields:!(ip,variable2),version:!t),index:%27as-*%27),title:%27New%20Saved%20Search1%27,type:search)",
            		"query": {
            			"jobParams": "(conflictedTypesFields:!(),fields:!(ip,variable2),indexPatternId:d58f2a10-ff77-11e7-ae73-fd89243a1cfc,metaFields:!(_source,_id,_type,_index,_score),searchRequest:(body:(_source:(excludes:!(),includes:!(ip,variable2)),docvalue_fields:!(),query:(bool:(filter:!(),must:!((match_all:())),must_not:!(),should:!())),script_fields:(),sort:!((_score:(order:desc))),stored_fields:!(ip,variable2),version:!t),index:'as-*'),title:'New Saved Search1',type:search)"
            		},
            		"pathname": "/api/reporting/generate/csv",
            		"path": "/api/reporting/generate/csv?jobParams=(conflictedTypesFields:!(),fields:!(ip,variable2),indexPatternId:d58f2a10-ff77-11e7-ae73-fd89243a1cfc,metaFields:!(_source,_id,_type,_index,_score),searchRequest:(body:(_source:(excludes:!(),includes:!(ip,variable2)),docvalue_fields:!(),query:(bool:(filter:!(),must:!((match_all:())),must_not:!(),should:!())),script_fields:(),sort:!((_score:(order:desc))),stored_fields:!(ip,variable2),version:!t),index:%27as-*%27),title:%27New%20Saved%20Search1%27,type:search)",
            		"href": "/api/reporting/generate/csv?jobParams=(conflictedTypesFields:!(),fields:!(ip,variable2),indexPatternId:d58f2a10-ff77-11e7-ae73-fd89243a1cfc,metaFields:!(_source,_id,_type,_index,_score),searchRequest:(body:(_source:(excludes:!(),includes:!(ip,variable2)),docvalue_fields:!(),query:(bool:(filter:!(),must:!((match_all:())),must_not:!(),should:!())),script_fields:(),sort:!((_score:(order:desc))),stored_fields:!(ip,variable2),version:!t),index:%27as-*%27),title:%27New%20Saved%20Search1%27,type:search)"
            	},
            	"message": "[remote_transport_exception] [node-2][IPNODE2:9300][indices:admin/create]"
}

I've tried playing with the settings with no result. Any ideas?

2

I think this is the key.

What version of Elasticsearch and Kibana are you running? Did you upgrade from some other version?

3

According to a developer familiar with reporting, the .reporting indices should only have documents with type esqueue in it. Do you have any idea how a document with type doc could have gotten into the index?

You could check how many documents are in the index in the Dev Tools > Console like;

GET .reporting-2018.01.21/_count

Then you can get the actual documents like;

GET .reporting-2018.01.21/_search

In my results after I generated 2 reports I see 2 documents with "_type": "esqueue"

image
image.png616×748 51.7 KB

If you don't need any reports that have been generated since 2018-01-21 you could delete that whole index and reporting would recreate it the next time you generate a report.

Or, you could delete the documents with "_type": "doc" but that would take a minute for me to figure out and test so let me know if you need to keep some existing reports.

Actually, now that I think about it, you can't have any actual reports in that index because they all would have gotten that error. So I expect it would be save to delete the whole index.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.