Monitor uptime is not shown

I was just too late to reply to this topic: Monitor uptime is not shown

So I'll reply in this new topic:

Hi @emilioalvap,

I tried the query you provided and I do get hits for all the browser monitors that I expect.

I did some searching and managed to log the query that elasticsearch is receiving to get the total up and down count:

{
    "size": 0,
    "query":
    {
        "match_none":
        {
            "boost": 1.0
        }
    },
    "aggregations":
    {
        "counts":
        {
            "scripted_metric":
            {
                "init_script":
                {
                    "source": "state.locStatus = new HashMap(); state.totalDocs = 0;",
                    "lang": "painless"
                },
                "map_script":
                {
                    "source": "\n          def loc = doc[\"observer.geo.name\"].size() == 0 ? \"\" : doc[\"observer.geo.name\"][0];\n\n          // One concern here is memory since we could build pretty gigantic maps. I've opted to\n          // stick to a simple <String,String> map to reduce memory overhead. This means we do\n          // a little string parsing to treat these strings as records that stay lexicographically\n          // sortable (which is important later).\n          // We encode the ID and location as $id.len:$id$loc\n          String id = doc[\"monitor.id\"][0];\n          String idLenDelim = Integer.toHexString(id.length()) + \":\" + id;\n          String idLoc = loc == null ? idLenDelim : idLenDelim + loc;\n\n          String status = doc[\"summary.down\"][0] > 0 ? \"d\" : \"u\";\n          String timeAndStatus = doc[\"@timestamp\"][0].toInstant().toEpochMilli().toString() + status;\n          if(state.locStatus[idLoc] == null){\n            state.locStatus[idLoc] = timeAndStatus;\n          }else if(timeAndStatus.compareTo(state.locStatus[idLoc]) > 0){\n            state.locStatus[idLoc] = timeAndStatus;\n          }\n\n          state.totalDocs++;\n        ",
                    "lang": "painless"
                },
                "combine_script":
                {
                    "source": "\n          return state;\n        ",
                    "lang": "painless"
                },
                "reduce_script":
                {
                    "source": "\n          // Use a treemap since it's traversable in sorted order.\n          // This is important later.\n          TreeMap locStatus = new TreeMap();\n          long totalDocs = 0;\n          int uniqueIds = 0;\n          for (state in states) {\n            totalDocs += state.totalDocs;\n            for (entry in state.locStatus.entrySet()) {\n              // Update the value for the given key if we have a more recent check from this location.\n              locStatus.merge(entry.getKey(), entry.getValue(), (a,b) -> a.compareTo(b) > 0 ? a : b)\n            }\n          }\n\n          HashMap locTotals = new HashMap();\n          int total = 0;\n          int down = 0;\n          String curId = \"\";\n          boolean curIdDown = false;\n          // We now iterate through our tree map in order, which means records for a given ID\n          // always are encountered one after the other. This saves us having to make an intermediate\n          // map.\n          for (entry in locStatus.entrySet()) {\n            String idLoc = entry.getKey();\n            String timeStatus = entry.getValue();\n\n            // Parse the length delimited id/location strings described in the map section\n            int colonIndex = idLoc.indexOf(\":\");\n            int idEnd = Integer.parseInt(idLoc.substring(0, colonIndex), 16) + colonIndex + 1;\n            String id = idLoc.substring(colonIndex + 1, idEnd);\n            String loc = idLoc.substring(idEnd, idLoc.length());\n            String status = timeStatus.substring(timeStatus.length() - 1);\n\n            // Here we increment counters for the up/down key per location\n            // We also create a new hashmap in locTotals if we've never seen this location\n            // before.\n            locTotals.compute(loc, (k,v) -> {\n              HashMap res = v;\n              if (v == null) {\n                res = new HashMap();\n                res.put('up', 0);\n                res.put('down', 0);\n              }\n\n              if (status == 'u') {\n                res.up++;\n              } else {\n                res.down++;\n              }\n\n              return res;\n            });\n\n\n            // We've encountered a new ID\n            if (curId != id) {\n              total++;\n              curId = id;\n              if (status == \"d\") {\n                curIdDown = true;\n                down++;\n              } else {\n                curIdDown = false;\n              }\n            } else if (!curIdDown) {\n              if (status == \"d\") {\n                curIdDown = true;\n                down++;\n              } else {\n                curIdDown = false;\n              }\n            }\n          }\n\n          Map result = new HashMap();\n          result.total = total;\n          result.location_totals = locTotals;\n          result.up = total - down;\n          result.down = down;\n          result.totalDocs = totalDocs;\n          return result;\n        ",
                    "lang": "painless"
                }
            }
        }
    }
}

It and it is only run for the following indexes:

  • .ds-synthetics-browser.network-default-2023.05.17-000002
  • .ds-heartbeat-8.6.1-2023.04.14-000001

And not for .ds-synthetics-browser-default-2023.05.17-000002 that I expect to be used.

Any idea why it is not checking all indexes?

Kind regards,
Sam Verhoeven

No this issue is not resolved yet. I only found out which query is being used, and that it's not what I would expect. But no idea why.

Apologies for the delay here.

It's a bit surprising, there's certainly something funky here, but whatever it is seems pretty specific to your setup in a way that's hard to understand. It seems to me like somehow datastreams aren't being setup correctly in your cluster, and that the alias for synthetics-* isn't matching that index.

You might try reproing with a brand ES cluster and Kibana. It's possible a one-off administrative action changed some aspect of ES ILM or similar.

Another thought, have you considered using the new Synthetics app instead? You can configure monitors either via the GUI or via the CLI with project monitors. The docs can be found here: Get started | Elastic Observability [8.8] | Elastic . It's in many ways a more powerful experience than plain heartbeat, and also provides great ergonomics if IaaC is important to you.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.