Namespaced Synthetic monitors are missing after upgrade

Hello everyone, after upgrading from 8.7.1 to 8.9.1 - I'm no longer able to see any of my monitored hosts but one from heartbeat in Kibana.
I can confirm the data is there by looking at the discover but no hosts are showing up in either Uptime -> Uptime Monitors or Synthetics -> Monitors.
I did notice 'Monitors in Synthetics may stop running' notice in changelog for 8.8.0, and I've logged in as user which has permissions to trigger updates in background but i still don't see any of my monitors.

I've also created new monitor from UI but then i can only see the new one, not any of my previously generated monitors.

The only one monitor that is available - it doesn't have namespace configured so it goes to 'default' namespace. It's configuration:

- type: http
  id: aws-node1-name
  name: Node 1 Name
  enabled: true
  schedule: '@every 60s'

  # Configure URLs to ping
  hosts: [""]

    method: "GET"
      "Entity-Protocol": "1"

    status: 200

And example of monitor with namespace (i have many of them, i will post one example though)

- type: http
  id: aws-node2-name
  enabled: true
  schedule: '@every 60s'
  data_stream.namespace: "aws_uptime"

  # Configure URLs to ping
  hosts: [""]

    status: 302

I did try to remove/comment data_stream configuration, forcing the heartbeat to use default namespace - and it showed up. Issue is only with namespaced monitors.

Previously (in 8.7.1), when no data_stream.namespace config is used, heartbeat will place all data in heartbeat-8.7.1 index (or now in heartbeat-8.9.1), but for those monitors that have data_stream.namespace configured - it went to synthetics-http-<namespace>.
It's still doing that but then it appears that some issue is preventing previously configured monitors to appear due to Kibana new way of adding monitors.
Since only monitors added directly via Kibana are visible - and none of previously defined.

I'm not sure how do i resolve this but it's critical since alerts doesn't seem to be working either. Alerts that are tied to those monitors.
Alerts have error message:

There has been a catastrophic error trying to install index level resources for the following registration context: observability.uptime. This may have been due to a non-additive change to the mappings, removal and type changes are not permitted. Full error: Error: Failure during installation of common resources shared between all indices. Timeout: it took more than 1200000ms

Hello @Marko_Todoric ,

Can you please verify what does your uptime setting index patterns contains, does it contains synthetics-*?

You can add that manually to the settings. We had a migration to remove synthetics-* by default from uptime app, since that data now appears in new synthetics app. but if you want to bring it back you can add that index manually. There was a realease note item in kibana regarding this.

Best Regards

Awsesome!!! Thank you very much. I've read release notes but I must've missed that one somehow.

Thanks a lot!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.