Viewing Metrics -- 500 Internal Server Error

Elastic Observability Metrics
1

I'm using ELK 7.6.1, all running on the same physical node as docker containers, launched via a docker-compose file. The only shipper setup so far is Filebeat for sending system logs. I can view these in Kibana. The problem is, when I try to bring up the Metrics page in Kibana UI, I am getting a pop-up in the lower right side of screen saying

Error while fetching resource
Error: Internal Server Error (500)
URL: http://instinct.nic.uoregon.edu:5601/api/metrics/snapshot

This is my browser console error:

POST http://instinct.nic.uoregon.edu:5601/api/metrics/snapshot 500 (Internal Server Error)

commons.bundle.js:3 Uncaught (in promise) Error: Internal Server Error
    at fetchResponse$ (commons.bundle.js:3)
    at s (kbn-ui-shared-deps.js:338)
    at Generator._invoke (kbn-ui-shared-deps.js:338)
    at Generator.forEach.e.<computed> [as next] (kbn-ui-shared-deps.js:338)
    at s (kbn-ui-shared-deps.js:338)
    at t (kbn-ui-shared-deps.js:338)
    at kbn-ui-shared-deps.js:338

Server log

$> docker logs -f elk_kibana_1 | grep 500
...
{"type":"response","@timestamp":"2020-03-28T17:04:17Z","tags":["access:infra"],"pid":6,"method":"post","statusCode":500,"req":{"url":"/api/metrics/snapshot","method":"post","headers":{"host":"instinct.nic.uoregon.edu:5601","connection":"keep-alive","content-length":"193","kbn-version":"7.6.1","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36","content-type":"application/json","accept":"*/*","origin":"http://instinct.nic.uoregon.edu:5601","referer":"http://instinct.nic.uoregon.edu:5601/app/infra","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9"},"remoteAddress":"73.96.133.171","userAgent":"73.96.133.171","referer":"http://instinct.nic.uoregon.edu:5601/app/infra"},"res":{"statusCode":500,"responseTime":135,"contentLength":9},"message":"POST /api/metrics/snapshot 500 135ms - 9.0B"}

Any ideas why I am seeing this only on the Metrics page? There are no metrics being shipped, but I imagine that should not be a reason for an ISE 500. Kibana is otherwise working as I would expect. I am new to ELK so totally possible I am doing something dumb.

1 Like
2

Hi @eugeneswalker,

thanks for sharing the detailed error messages. I suspect the underlying reason should be shown in the Kibana server log. What you posted as the server log looks like reverse proxy's access log. Can you get at the log output of the Kibana server process somehow?

3

I have the exact same error, I'll try to do what you had which was using 7.6.1 to see if it will start working again. In my case I am running behind a proxy so I set the following environment variables

- "SERVER_BASEPATH=/kibana"
- "SERVER_REWRITEBASEPATH=true"
- "LOGGING_SILENT=true"

Unfortunately 7.6.1 didn't help either. BTW I didn't get anything on the logs that appear relevant even with LOGGING_SILENT not set

What worked in the end was downgrading Kibana to 7.5.2

4

Hi, I have had the same problem for two weeks. Every time I wanted to see the metrics of a specific host, it returned an error 500.

Today I solved it by configuring apm-server, enabling it in kibana (I had disabled it because I thought I would not use it) and allowing auto-creation of the apm- * index in elasticsearch.

Now I can see the metric detail for each node.

If you need I can share my elasticsearch conf, kibana, metricbeat and apm-server.

5

Can you share the config files please? Thank's

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.