Hello everyone, after upgrading from 8.7.1 to 8.9.1 - I'm no longer able to see any of my monitored hosts but one from heartbeat in Kibana.
I can confirm the data is there by looking at the discover but no hosts are showing up in either Uptime -> Uptime Monitors or Synthetics -> Monitors.
I did notice 'Monitors in Synthetics may stop running' notice in changelog for 8.8.0, and I've logged in as user which has permissions to trigger updates in background but i still don't see any of my monitors.
I've also created new monitor from UI but then i can only see the new one, not any of my previously generated monitors.
The only one monitor that is available - it doesn't have namespace configured so it goes to 'default' namespace. It's configuration:
- type: http
id: aws-node1-name
name: Node 1 Name
enabled: true
schedule: '@every 60s'
# Configure URLs to ping
hosts: ["https://url.to.example.org/app/path?ping"]
check.request:
method: "GET"
headers:
"Entity-Protocol": "1"
check.response:
status: 200
And example of monitor with namespace (i have many of them, i will post one example though)
- type: http
id: aws-node2-name
name: node2.example.org
enabled: true
schedule: '@every 60s'
data_stream.namespace: "aws_uptime"
# Configure URLs to ping
hosts: ["https://node2.example.org"]
check.response:
status: 302
I did try to remove/comment data_stream configuration, forcing the heartbeat to use default namespace - and it showed up. Issue is only with namespaced monitors.
Previously (in 8.7.1), when no data_stream.namespace config is used, heartbeat will place all data in heartbeat-8.7.1 index (or now in heartbeat-8.9.1), but for those monitors that have data_stream.namespace configured - it went to synthetics-http-<namespace>.
It's still doing that but then it appears that some issue is preventing previously configured monitors to appear due to Kibana new way of adding monitors.
Since only monitors added directly via Kibana are visible - and none of previously defined.
I'm not sure how do i resolve this but it's critical since alerts doesn't seem to be working either. Alerts that are tied to those monitors.
Alerts have error message:
There has been a catastrophic error trying to install index level resources for the following registration context: observability.uptime. This may have been due to a non-additive change to the mappings, removal and type changes are not permitted. Full error: Error: Failure during installation of common resources shared between all indices. Timeout: it took more than 1200000ms
