Feedback regarding Synthetics Tests

Elastic Observability Synthetics
6

Dear team,

Thank you so much for the links - I already subscribed the Github issues to be informed about any news and I am happy that our main problem regarding privileges might be solved in 8.9 :+1:

We are progressing in regards to our tests and we found a few other (but only minor) issues. Although some of them may be related to playwright and not the Elastic Synthetics package itself, I will show them here for reference:

  1. Different behaviour between headless and interactive mode
    I am a german user and when running the Synthetics Recorder, the application is in german. This is also true when running synthetics with --playwright-options '{"headless": false}', but when running the tests in headless mode, the monitor suddenly fails. After digging around, I found that in headless mode, the application suddenly shows in english. I solved this by setting the locale in the playwrightOptions, but I think it would make sense to have the same behaviour in all environments.

  2. Minor code generation issue for downloads
    When I use the recorder and click on a download, it generates code like this:

const downloadPromise = page.waitForEvent('download');
await page.getByRole('button', { name: 'Excel Export' }).click();
download = await downloadPromise;

The error at runtime is:

 stack: |-
   ReferenceError: download is not defined
       at Step.callback (D:\Dev\WORKSPACE_NEU\tcmore-test\journeys\example.journey.ts:16:5)
       at Runner.runStep (D:\Dev\WORKSPACE_NEU\tcmore->test\node_modules\@elastic\synthetics\src\core\runner.ts:212:7)
       at Runner.runSteps (D:\Dev\WORKSPACE_NEU\tcmore->test\node_modules\@elastic\synthetics\src\core\runner.ts:262:16)
       at Runner.runJourney (D:\Dev\WORKSPACE_NEU\tcmore->test\node_modules\@elastic\synthetics\src\core\runner.ts:352:27)
       at Runner.run (D:\Dev\WORKSPACE_NEU\tcmore->test\node_modules\@elastic\synthetics\src\core\runner.ts:445:11)
       at Command.<anonymous> (D:\Dev\WORKSPACE_NEU\tcmore->test\node_modules\@elastic\synthetics\src\cli.ts:137:23)

To fix this, I had to add the var keyword:

const downloadPromise = page.waitForEvent('download');
await page.getByRole('button', { name: 'Excel Export' }).click();
var download = await downloadPromise;
  1. Use import instead of require for project monitors
    Currently, the Recorder generates the following statement for project monitors:
const { journey, step, expect } = require('@elastic/synthetics');

This works fine and as long as you have only one journey-file, this is also not a problem in other tools. But when having multiple journey files and using tools like VS Code you get the following error:

Cannot redeclare block-scoped variable 'journey'. ts(2451)

This can be fixed by rewriting it to:

import { journey, step } from "@elastic/synthetics";

Is it possible that the recorder generates an import-statement instead of require?

  1. Wrong code generation on navigation
    A few times - but not always - the Recorder generated wrong code when submitting a form. I was not able to reproduce in which situations this happens.
    Instead of:
journey('Recorded journey', async ({ page, context }) => {
  step('Go to application', async () => {
    await page.goto('https://my-host/prod/my-app/login.html');
    await page.locator('#inputUname').click();
    await page.locator('#inputUname').fill('user');
    await page.locator('#inputUname').press('Tab');
    await page.locator('#inputPwd').fill('S3cr3t!');
    await page.locator('#inputPwd').press('Enter');
  });
});

The Recorder sometimes generates:

journey('Recorded journey', async ({ page, context }) => {
  step('Go to application', async () => {
    await page.goto('https://my-host/prod/my-app/login.html');
    await page.locator('#inputUname').click();
    await page.locator('#inputUname').fill('user');
    await page.locator('#inputUname').press('Tab');
    await page.locator('#inputPwd').fill('S3cr3t!');
    await page.goto('https://my-host/prod/my-app/index.html');
  });
});

So, instead of submitting the form to login, it only fills the form and then tries to visit the application homepage which requires authentication.

Best regards
Wolfram

PS - cleaning up objects after Test

The following is neither a bug nor a feature request, but a bit of info for others (or maybe someone has a better idea?):
Not all applications are reporting tools where the Monitor only does readonly operations, there are also Applications where we may want to monitor the process of creating and/or modifying objects. In this usecase, the synthetic monitors may fail when the previous run did not correctly cleanup. To make the cleanup more resistant, I moved them out of the normal journey steps and put them in a cleanup step after the journey completed. This is my little cleanup.helpers.ts:

import { after } from "@elastic/synthetics";

/**
 * Cleanup Entry
 */
class CleanupEntry {
    /**
     * constructor
     * @param name the name of the object to cleanup
     * @param callback the function containing the cleanup logic
     */
    constructor(public name: string, public callback: () => Promise<void>) {}
  }
  
  /**
   * Contains all cleanup actions to process
   */
  var toClean: CleanupEntry[] = [];
  
  /**
   * add a cleanup action to run after the journey is complete
   * @param name the name of the object to cleanup
   * @param asyncCleanupFunction the function containing the cleanup logic
   */
  export function doCleanup(
    name: string,
    asyncCleanupFunction: () => Promise<void>
  ): void {
    toClean.unshift(new CleanupEntry(name, asyncCleanupFunction));
  }
  /**
   * register the cleanup process to run after the journey is complete
   * @param page the page object from the journey
   */
  export function registerCleanup(page): void {
    after(async ({ params }) => {
      for (let i = 0; i < toClean.length; i++) {
        try {
          console.log("Cleaning: " + toClean[i].name);
          await toClean[i].callback();
        } catch (e) {
          console.log(e);
        }
      }
    });
  }

This can now be used like:

journey("Recorded journey", async ({ page, context }) => {
  // Only relevant for the push command to create
  // monitors in Kibana
  monitor.use({
    id: "example-monitor",
    schedule: 10,
  });
  step("Login", async () => {
    await page.goto(
      "https://myhost/myapp/login.html"
    );
    await page.locator("#inputUname").click();
    await page.locator("#inputUname").fill("user");
    await page.locator("#inputUname").press("Tab");
    await page.locator("#inputPwd").fill("s3cr3t!");
    await page.locator("#inputPwd").press("Enter");
    doCleanup("Logout", async () => {
      await page
        .getByRole("link", { name: "Logout" })
        .click();
    });
  });
  step("Add User", async () => {
    //... create user code
    doCleanup("Delete User", async () => {
		//... delete user code
    });
  });
  step("Add Group", async () => {
    //... create group code
    doCleanup("Delete Group", async () => {
		//... delete group code
    });
  });
  registerCleanup(page);
});

When a step executes successfully, the doCleanup will register the cleanup code to be run at the end of the journey in reverse order. This allows to be the latter objects to be deleted before possible dependencies are deleted.
When a single cleanup callback fails, the others are still tried to cleanup as much as possible and the error is logged to console.
When a step in the journey fails, the according cleanup-job will not run.

Unfortunately, throwing an exception in the after-call does not fail the process because the journey steps themself were successfull. This means, that currently, a failed cleanup process will only be shown on the next run because the creation of the object will fail.