Kibana Metrics – 500 Internal Server Error

[tvander]

When i'm trying to access to the metrics page (https://kibana-elk.pxdtools.io:5601/app/metrics) on Kibana i have an error 500 showing on the right bottom corner, and i can't use this page, i've got the "There is no data to display." message.

I changed the log to verbose on kibana config but i dont have many information, i only have these two logs:

{ 
"type": "error", 
"@timestamp": "2020-06-22T12:46:03Z", 
"tags": [], 
"pid": 3685, 
"level": "error", 
"error": { 
"message": "Internal Server Error", 
"name": "Error", 
"stack": "Error: Internal Server Error\n at HapiResponseAdapter.toError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:132:19)\n at HapiResponseAdapter.toHapiResponse (/usr/share/kibana/src/core/server/http/router/response_adapter.js:86:19)\n at HapiResponseAdapter.handle (/usr/share/kibana/src/core/server/http/router/response_adapter.js:81:17)\n at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:160:34)\n at process._tickCallback (internal/process/next_tick.js:68:7)" 
}, 
"url": { 
"protocol": null, 
"slashes": null, 
"auth": null, 
"host": null, 
"port": null, 
"hostname": null, 
"hash": null, 
"search": null, 
"query": {}, 
"pathname": "/api/metrics/snapshot", 
"path": "/api/metrics/snapshot", 
"href": "/api/metrics/snapshot" 
}, 
"message": "Internal Server Error" 
} 

{ 
"type": "response", 
"@timestamp": "2020-06-22T12:18:19Z", 
"tags": [ 
"access:infra" 
], 
"pid": 3685, 
"method": "post", 
"statusCode": 500, 
"req": { 
"url": "/api/metrics/snapshot", 
"method": "post", 
"headers": { 
"host": "kibana-elk.pxdtools.io:5601", 
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0", 
"accept": "*/*", 
"accept-language": "fr,fr-FR;q=0.8,en-US;q=0.5,en;q=0.3", 
"accept-encoding": "gzip, deflate, br", 
"referer": "https://kibana-elk.pxdtools.io:5601/app/metrics/inventory?waffleFilter=(expression:%27%27,kind:kuery)&waffleTime=(currentTime:1592828234054,isAutoReloading:!f)&waffleOptions=(accountId:%27%27,autoBounds:!t,boundsOverride:(max:1,min:0),customMetrics:!(),customOptions:!(),groupBy:!(),metric:(type:cpu),nodeType:host,region:%27%27,view:map)", 
"content-type": "application/json", 
"kbn-version": "7.8.0", 
"origin": "https://kibana-elk.pxdtools.io:5601", 
"content-length": "191", 
"connection": "keep-alive" 
}, 
"remoteAddress": "10.242.2.4", 
"userAgent": "10.242.2.4", 
"referer": "https://kibana-elk.pxdtools.io:5601/app/metrics/inventory?waffleFilter=(expression:%27%27,kind:kuery)&waffleTime=(currentTime:1592828234054,isAutoReloading:!f)&waffleOptions=(accountId:%27%27,autoBounds:!t,boundsOverride:(max:1,min:0),customMetrics:!(),customOptions:!(),groupBy:!(),metric:(type:cpu),nodeType:host,region:%27%27,view:map)" 
}, 
"res": { 
"statusCode": 500, 
"responseTime": 91, 
"contentLength": 9 
}, 
"message": "POST /api/metrics/snapshot 500 91ms - 9.0B" 
} 

My problem seems similar to this one: Viewing Metrics -- 500 Internal Server Error
but there is no fix.

Here is the installation detail:
3 nodes elasticsearch with license/xpack/security installed
1 kibana on different server with security
1 separate logstash and many others filebeat/metricbeat clients.

all stack is running on elk 7.8

i can correctly see metricbeat data from discover page.

[ntk]

I'm experiencing the same issue (ELK 7.8):

    {"type":"error","@timestamp":"2020-06-23T14:29:01Z","tags":[],"pid":21238,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error    at HapiResponseAdapter.toError (/home/ubuntu/kibana/src/core/server/http/router/response_adapter.js:130:19)
    at HapiResponseAdapter.toHapiResponse (/home/ubuntu/kibana/src/core/server/http/router/response_adapter.js:84:19)
    at HapiResponseAdapter.handle (/home/ubuntu/kibana/src/core/server/http/router/response_adapter.js:79:17)
    at Router.handle (/home/ubuntu/kibana/src/core/server/http/router/router.js:162:34)
    at process._tickCallback (internal/process/next_tick.js:68:7)"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":null,"query":{},"pathname":"/api/metrics/snapshot","path":"/api/metrics/snapshot","href":"/api/metrics/snapshot"},"message":"Internal Server Error"}

[tvander]

Problem solved,

This problem appears because of the index template, in my case i use logstash as intermediate between metricbeat and elasticsearch, so it seems that the existing template wasn't good,

I had to manually export the template on my metricbeat client server and then manually import it
https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-template.html#load-template-manually-alternate

metricbeat export template > metricbeat.template.json
curl -k  -u elastic:mypass -XPUT -H 'Content-Type: application/json' https://PIXID-ELKMASTER1:9200/_template/metricbeat-new -d@metricbeat.template.json

Then i stopped logstash service, deleted the old metricbeat index template, deleted my metricbeat index,

When i started logstash the metricbeat index had correctly been created by the pipeline configuration, and i do no longer have the 500 error on the kibana metrics page.

Hope it will help