Is there anyway to run async code before the SyntheticsConfig object in synthetics.config.ts is created?

[ameindel]

Hello, Elastic!

Our company has a home-rolled module that makes calls to AWS Parameter Store to retrieve secrets.

I'd like to utilize this module in our Elastic Synthetics project. The usage would be making a call to the module's loadConfig() method right before the SyntheticsConfig object is created in synthetics.config.ts. Parameters retrieved from the call would be injected into the SyntheticsConfig class downstream and then picked up by the synthetics via the param parameter.

The only issue is that loadConfig has asynchronous code and the code that creates the SyntheticsConfig object (export default (env) => {}) is synchronous. I was wondering if there was an undocumented way to create SynethticsConfig in an asynchronous manner?

Any additional information regarding best practices for retrieving configuration data for synthetics would be welcomed as well.

Thank you in advance

Adam

[shahzad31 (Shahzad)]

Hello @ameindel ,

it's not possible at the moment to declare config file as async, i have created an issue in repo, we will discuss and see if it's worth making resolution async Enhancement: Make resolution of config file async as well · Issue #820 · elastic/synthetics · GitHub

Meanwhile i will suggest you maybe create a helper function which you can call before every journey in before step or maybe create a dedicated step for reading secrets.

Other way would be to use Synthetics app global params in UI. Those are auto injected into every journey.

Some examples of helper functions are here https://github.com/elastic/synthetics-demo/blob/main/todos/journeys/advanced-example-helpers.ts

I hope this helps.

Best Regards

[ameindel]

Hi, shahzad31!

Thank you for the prompt reply. The only issue I can see with embedding the loadConfig() logic in an async function at the synthetic level is that it would be making calls to AWS Parameter Store every time it runs. We will eventually have quite a number of synthetics and having each one hit PS every couple minutes seems like it might not be ideal, since the config data is only needed once.

That's great that you're open to discussing enhancing synthetics.config.ts to support async operations.

I'll also investigate the global params in the UI option as well as the helper function code you linked.

Thanks again!

Adam

[vigneshshanmugam]

Hi @ameindel ,

Thanks for raising this request.

I am going to keep the Async loading config support in the back, we will come back in a bit.

There are already couple of alternatives when it comes to supporting dynamic parameters when it comes to running vs pushing project based monitors.

1. Support via Global Parameters on the Synthetics UI, Applies for both browser and lightweight monitors.

2. Injecting parameter before calling push command or running monitors using the --param {} CLI flag.

// params.ts

async function fetchRemoteParams(){
  return { foo: "bar"}
}

fetchRemoteParams().then((params) => {
  process.env.REMOTE_PARAMS = params;
});

You can inject them at runtime to the CLI before calling Push/Run command like this

npx @elastic/synthetics push --params process.env.REMOTE_PARAMS 

3. Via the ENV variable in the Synthetics.config.ts file.

The previous way limits the ability to type things, if you want to support strong typings

export default (env) => {
  params: {
     secret: env["REMOTE_SECRET"]
  }
}

Any of these options would work and would not run in to the multiple calls limit that you have mentioned. But, looking at your use-case the easiest would be storing them ideally on the Global params as it makes it easier to support both Lightweight and Browser based monitors.

I would like to few other questions before we decide its worth supporting async fetching of the config itself.

1. Are you interested in fetching the config for managing the monitors on the Synthetics Kibana UI or for running the tests locally on a CI or Dev machine.
2. Are these remote secrets used across all monitors including lightweight monitors if you are planning to have few?

Thanks in advance and Please let us know if you have any other questions or feedback.

[ameindel]

Hi, @vigneshshanmugam ,

Thank you for the very informative reply. We are also considering a 4th options which would decrypt an encrypted secret that is set in synthetics.config.ts, effectively bypassing the need to call Parameter Store altogether. So essentially:

const config: SyntheticsConfig = {
  params : {
    url: "https://ourcompanysurl.com",
    hostName: 'Dev-Next Provider',
    team: "Observability",
    mailosaurSynthetic : {
	  key: "qwerty",
	  domain: "random.mailosaur.net",
	  account: "1234567890",
	  setting: decrypt('U2FsdGVkX1970jxPILRjjCOF3BOYT281RQUGHShB71s=')
    },
},
...
// String "It Works!" encrypted value is "U2FsdGVkX1970jxPILRjjCOF3BOYT281RQUGHShB71s="

Using this helper function:

const decrypt = (cipher) => {
  let phrase: any = process.env.PHRASE;
  const bytes = AES.decrypt(cipher, phrase);
  const originalText = bytes.toString(Utf8);
  return originalText;
}

Plugged into a journey:

step('Test decrypting a secret string', async () => {
  console.log('Setting: ' + params.mailosaurSynthetic.setting);
  await page.goto(params.url);
});

The output:

Journey: Test decrypting a secret string

Setting: It worked!

That being said, it still would be nice to leverage existing modules in an async fashion, if need be. To answer your questions:

1. We are interested in both of those scenarios. If it wasn't already apparent, I should mention that we house our current synthetics in a project (under Gitlab source control). Any engineer interested in developing synthetics can clone the project and test locally before commiting any new changes. We have a .gitlab-ci.yml pipeline that uses the push command to deploy to Kibana instances in our various environments. The --param option looks like it would be very handy for this.

2. Some are and some aren't. We have synthetics that use the same configuration data and some synthetics that have their own configuration data that no other synthetic uses.

Hope that helps!

I should mention that we're not 100% sure we're going with implementation #4. I'm going to keep playing around with it, but it looks like a promising, easy to implement, non-async alternative to keeping secret strings secret.

[vigneshshanmugam]

Hi @ameindel

Thanks for the details. Its definitely helpful.

We are working on supporting the Async fetching of information inside the Config file, which would be available in the following releases. Please watch out this enhancement issue for more details.

For the time being, any of the options proposed or the 4th option you mentioned would work just fine.

As its just JS, you could also just fetch the file on exec and just use the params

const remoteParams = require("./fetch-params");

export default env => {
  params: {
      "foo": "bar",
      ...remoteParams
   } 
}

Thanks,
Vignesh

[shahzad31 (Shahzad)]

We merged a PR feat: read config file async by shahzad31 · Pull Request #823 · elastic/synthetics · GitHub
it will be available in next release.

Best Regards