We are currently evaluating the Synthetic Monitoring of Elastic with 2 applications (1 Java with JSP, 1 Angular) and although we did not create huge tests yet, I would like to give our opinion about the current status:
Overall, I like the experience of running and monitoring the tests already while the recorder is still in an early stage and misses a few features that would make it really useful. The 2 things I like the most are the project monitors to support version control and the detailed overview over what is happening in each step (like the screenshots and the detailed network requests).
With this being said, here is a list of features that would greatly improve the experience in my opinion:
My biggest problem are the currently required privileges for creating Synthetic Monitoring Tests on a private location as I need to grant
all privileges on fleet and integrations to all spaces when a user should be able to push tests for a SINGLE space on a SINGLE private location. With this, he would be able to modify private locations, edit integrations other than synthetic monitoring and editing all other fleet agents and he also sees all spaces instead of ONE space he has access to. So basically, I would either grant them full access to fleet or I would have to manage the tests of all users for them.
As I am already using Elastic APM I configured the APM service name for my tests, but I was unable to find a link within the synthetic test details to show the corresponding APM traces. It would be great to have a direct link to APM (and of course other features like logs and metrics).
When using custom certificates with our own root CA, synthetic tests could use a bit of improvement. You can either disable the certificate validation (which is something nobody should ever do), or you can create your custom image based on the official Elastic agent image with your own certificates included. We currently use option 2 but it would be cool not having to maintain our own version of the image.
Another improvement would be to check the certificates on servers checked with synthetic tests the same way as it is already done with HTTP single page tests. Of course, this can currently be solved with a dummy HTTP test against the server but it would be great to support it nevertheless.
I don't know if this is possible but I would like to be able to run the synthetic tests in a CI pipeline: When using project monitors, the tester or developer could push changes to the tests into GIT, the CI would run the tests and fail if any of the tests fail. This way, only correct tests would be merged into the main branch. Maybe this could be possible with a special command option to the elastic agent image?
When running the Test with the synthetics recorder, the Browser window will be closed directly after completion. It would be great if the browser window would stay open after completion: For failed tests, we could check what happened and fix the test before a restart. For successfull tests, we could then add additional actions/steps.
It would be cool if the recorder had the option to reorder actions within a step.
Currently, the selector is already shown when hovering over an element in the browser. When adding an assertion, we need to manually type this selector into the recorder. It would be great if we could either copy the selector for the current element to paste it into the recorder or if we could select an element from within the recorder (like the element selector feature in the browser DevTools).
It would be a good addition to allow access to settings like timeouts and so on in the recorder (which can be configured using the playwright options in Kibana).
It would be great to have parameter support in the recorder. This way, we could already create the parameters there and prevent hardcoded entries from being pushed to Elastic and Git. This is especially useful for not leaking passwords.
When an application uses OIDC (e.g. with Keycloak), the synthetics recorder detects the redirect as a navigation and adds this to the test. Although it is easy to delete the step, maybe it is possible to detect such redirects and ignoring them? Maybe it even makes sense to ignore all redirects as they are sent by the server and we expect the server to send them for all runs?