Get synthetic monitor data: how?

[NominaSumpta (William Edwards)]

Hi,

How do I get synthetic monitor data shown in Kibana from an API?

Kibana provides cool monitor visualisations and data, such as 'Last 10 Test Runs' (table), 'Duration trends' (graphs), etc.

There doesn't seem to be a high-level way of retrieving this data from Kibana (at least, not according to Synthetics APIs | Kibana Guide [8.14] | Elastic).

The logical go-to is re-constructing such tables/graphs myself using 'raw' data from Elasticsearch, but I can't tell the query that Kibana used in the first place; when navigating to Observability -> Explore data, I can open an element in Lens, but no KQL syntax is shown.

Also, the shown data types (such as the 'Monitor duration' 'report metric' are Observability-specific:

... so I'm not sure how I would refer to such data types in an API call.

In Lens, I see that a temporary data view is used:

... but opening it doesn't yield any useful information either, except for that Synthetics data is stored in the synthetics-http-default data stream:

... which brings me back to pulling data out of Elasticsearch directly.

I hope someone can clear up the confusion!

[Alex_Salgado-Elastic (Alex Salgado)]

@NominaSumpta

... but opening it doesn't yield any useful information either, except for that Synthetics data is stored in the synthetics-http-default data stream:

Hello, have you tried first select your data using Dev Tools?

[NominaSumpta (William Edwards)]

Hi Alex,

After some more digging, I found I can get to Dev Tools (and find the Elasticsearch query used to construct several 'widgets') by opening the options menu:

... followed by 'More' -> 'Inspect' -> 'View: Data' -> 'Requests' -> 'Request'.

Furthermore, I saw that Kibana does API calls to endpoints such as /internal/synthetics/pings. They're internal, so not meant to be used by systems other than Kibana: [edited] Calls to Kibana internal APIs require an internal product header · Issue #152287 · elastic/kibana · GitHub. However, I can derive from them the other Elasticsearch requests: kibana/x-pack/plugins/observability_solution/synthetics/server/routes/pings/get_ping_statuses.ts at 252e035594a4c680cdbf8ef4a1009c62c0ed76a1 · elastic/kibana · GitHub -> kibana/x-pack/plugins/observability_solution/synthetics/server/common/pings/query_pings.ts at 252e035594a4c680cdbf8ef4a1009c62c0ed76a1 · elastic/kibana · GitHub

This begs the question: is the structure of Observability documents stable? In other words: can I get these documents from a non-Kibana system, assuming my implementation won't break?

The same goes for APM.

[Alex_Salgado-Elastic (Alex Salgado)]

Hi @NominaSumpta

If I understood your question correctly, yes, all the information collected is stored in Elasticsearch which, when properly designed, is super robust, scalable, and secure: https://www.elastic.co/elasticsearch/features.

As for the information being accessed externally, by creating an API_key for security reasons, you can access the data like any other index, as in this example of the synthetic-browser-default data:

`
curl -X POST "${ES_URL}/.ds-synthetics-browser-default-2024.06.09-000001/_search?pretty"
-H "Authorization: ApiKey "${ES_API_KEY}""
-H "Content-Type: application/json"
-d'
{
"query": {
"match_all": {}
}
}
'

`

Feel free to continue asking your questions here, share your results, and I suggest creating a separate post for small questions to make the most out of the community's responses.

[NominaSumpta (William Edwards)]

Hi Alex,

I think I was unclear by using the word 'stable'. By 'stable', I'm not talking about the ecosystem or the ELK stack itself, but about the format of Observability documents (those indexed in Elasticsearch). For example, if fields or the contents of fields suddenly change, my consumers would break.

[Alex_Salgado-Elastic (Alex Salgado)]

@NominaSumpta

I think I was unclear by using the word 'stable'. By 'stable', I'm not talking about the ecosystem or the ELK stack itself, but about the format of Observability documents (those indexed in Elasticsearch). For example, if fields or the contents of fields suddenly change, my consumers would break.

Hi William,

Elastic does a good job of maintaining backward compatibility, even to avoid cases like the one you mentioned. However, it's important to actively manage and monitor your schema, and keep up with the evolution of OpenTelemetry to ensure stability for your consumers.

This is my point of view, but I suggest you open a new thread with this question to possibly get information and facts from the development team itself.

[NominaSumpta (William Edwards)]

Done: Backward compatibility / stability of Observability documents