Monitoring cluster logstash error

[bevano (Bevan)]

Hi all,

I currently have a ELK stack in product and have metricbeat setup on every node logstash instances pushing metric and monitoring data to a separate monitoring cluster.

Everything was working in 7.7.1, but since updating to 7.8 when i try and open the information tab on a specific logstash node, i receive a {"statusCode":404,"error":"Not Found","message":"Not Found"}

The information tab on a specific logstash node is that tab where you can see the persistent queue and jvm heap performance. Metricbeat is working as expected, i can see the systems CPU/Memory usage, and also the Events Ingested counter works.

Not sure whether this is a bug, or if I can fix it.

This is the logstash overview page that is working

[bevano (Bevan)]

This is from the system logs

Jun 26 17:05:08 ELKMONITOR kibana[959]: {"type":"response","@timestamp":"2020-06-26T07:05:07Z","tags":[],"pid":959,"method":"get","statusCode":404,"req":{"url":"/logstash/node/abb48d76-112a-4fde-9f12-6d1cc08dedf6?_g=%28cluster_uuid%3AmVxn9bzlQACfBH1ExjHAPA%2CrefreshInterval%3A%28pause%3A%21f%2Cvalue%3A10000%29%2Ctime%3A%28from%3Anow-10m%2Cto%3Anow%29%29","method":"get","headers":{"host":"192.168.0.XXX:5601","connection":"keep-alive","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","accept-encoding":"gzip, deflate","accept-language":"en-GB,en-US;q=0.9,en;q=0.8"},"remoteAddress":"10.210.0.77","userAgent":"10.210.0.77"},"res":{"statusCode":404,"responseTime":120,"contentLength":9},"message":"GET /logstash/node/abb48d76-112a-4fde-9f12-6d1cc08dedf6?_g=%28cluster_uuid%3AmVxn9bzlQACfBH1ExjHAPA%2CrefreshInterval%3A%28pause%3A%21f%2Cvalue%3A10000%29%2Ctime%3A%28from%3Anow-10m%2Cto%3Anow%29%29 404 120ms - 9.0B"}

[christian.kordein (Chris)]

Facing the same error after Upgrading and the Information Tab from the pipelines is also not showing here (Same 404 Message).

One thing i found was in Kibana Logs:

{"type":"log","@timestamp":"2020-06-26T13:04:48Z","tags":["debug","http","server","Kibana","cookie-session-storage"],"pid":21851,"message":"Error: Unauthorized"}
{"type":"log","@timestamp":"2020-06-26T13:04:48Z","tags":["debug","plugins","security","basic","basic"],"pid":21851,"message":"Trying to authenticate user request to /api/status."}
{"type":"log","@timestamp":"2020-06-26T13:04:48Z","tags":["debug","plugins","security","basic","basic"],"pid":21851,"message":"Cannot authenticate requests with `Authorization` header."}

Hopefully someone can support.

[Wolfram_Haussig]

I have compared the urls from the working pipeline overview and the not working pipeline details:
https://host:5601/app/monitoring#/logstash/pipelines?_g=(cluster_uuid:VTDsWXH0TUqC4YO7H8Zhkw,refreshInterval:(pause:!f,value:10000),time:(from:now-1h,to:now))
https://host:5601/logstash/pipelines/central-syslog?_g=(cluster_uuid:VTDsWXH0TUqC4YO7H8Zhkw,refreshInterval:(pause:!f,value:10000),time:(from:now-1h,to:now))

The problem is that the part "/app/monitoring#" is missing in the lower url. adding this part produces the correct url

Update: Problem will be fixed with 7.8.1 https://github.com/elastic/kibana/pull/69861

[bevano (Bevan)]

Great, thanks for the update.