Elastic Synthetics Journey: Receiving `permission denied`

[DougR]

@DougR

This is a simple browser test that I'm using and which is stuck in pending. It succeeds when run locally.

import { journey, step, monitor, expect } from '@elastic/synthetics';

journey('Google Test', ({ page, params }) => {
  monitor.use({
    id: '2252824d-ba39-454e-8ebd-d749de23279f',
    schedule: 10,
  });

  step('Open https://google.com', async () => {
    await page.goto('https://google.com');
    await expect(page.getByText('Google')).toBeTruthy();
  });
});

Updates

Update #1

When I create a multi-step journey in the UI and paste this script in, it fails, but it does attempt to run.

Update #2

I figured out what the issue was and it now runs as expected when I create a multi-step journey with the above script in the UI. Still not working when pushed from CLI.

[DougR]

I found the following in the v8.7 documentation:

Synthetic tests cannot run under the root user. Refer to Synthetics Fleet Quickstart for more information.

I have to wonder whether this is the issue. When I look at the processes running in the container, they're all running as root. Are all synthetics test pushed through the UI automagically run as elastic-agent? OTOH, if this were the issue, I would expect the lightweight tests I'm pushing via @elastic/synthetics to also fail.

@Andrew_Cholakian1 - Since your deployment is working as expected, can you provide any insight?

[DougR]

When I attempt to deploy a browser test to an on-prem node (e.g., an Elastic Agent deployed to a Linux server, rather than to an elastic-agent-complete container), I get the following error:

[tf4 (Thomaz)]

I'm also stuck into the same error. Browser monitors run correctly when created through Kibana but they don't when created using project monitors.

I tried changing the user of my k8s container to "elastic-agent" as described in the docs but it gives me other permission errors while starting up the container.

[DougR]

@tf4

I tried changing the user of my k8s container to "elastic-agent" as described in the docs but it gives me other permission errors while starting up the container.

Same here.

[Andrew_Cholakian1 (Andrew Cholakian)]

Sorry for the delay here, trying to balance a number of priorities.

@tf4 what are the errors you're getting?

We're probably a week or so out from being able to dedicate some time to look into this unfortunately.

[tf4 (Thomaz)]

@Andrew_Cholakian1 I'm getting exactly the same error that @DougR mentioned in the beginning of this thread. BTW I'm using ECK and elastic-agent-complete:8.7.0 as well.

When I change the user to elastic-agent (runAsUser: 1000) I keep getting the following error and the container doesn't start.

cp: cannot create regular file '/usr/local/share/ca-certificates/ca.crt': Permission denied

I know this is related to the default volume mounts of the agent but I believe I shouldn't change anything in those volumes.

[tf4 (Thomaz)]

In my last investigation I checked the permissions of the folders inside /tmp and discovered that my monitors are being copied to the agent with permissions to the root user only. I suspect this is the reason why the elastic-agent user can't execute that .bin/elastic-synthetics command.

[emilioalvap (Emilio Alvarez Piñeiro)]

Hi @DougR, @tf4,

Elastic agent cannot run as non-root user, it's a known issue for ECK deployments. Any errors you get when changing the user are probably related to that.

Could you check under what user and capabilities heartbeat process is running? What does the env variable BEAT_SETUID_AS evaluate to inside the container?

We have mechanisms in place to prevent running browser monitors as a root user which are probably interfering here. FYI, we have introduced a new approach on 8.7.1.

[tf4 (Thomaz)]

Hi @emilioalvap!

Thanks for the info about running as non-root user.

As you can see in the output below, there are two heartbeat processes running as root.

root       924  0.1  0.5 1503900 88436 ?       Sl   13:22   0:01 /usr/share/elastic-agent/data/elastic-agent-10dc6a/components/heartbeat -E setup.ilm.enabled=false -E setup.template.enabled=false -E management.enabled=true -E management.restart_on_output_change=true -E logging.level=info -E logging.to_stderr=true -E gc_percent=${HEARTBEAT_GOGC:100} -E http.enabled=true -E http.host=unix:///usr/share/elastic-agent/state/data/tmp/synthetics-http-default.sock -E path.data=/usr/share/elastic-agent/state/data/run/synthetics/http-default
root       942  0.2  0.5 2565076 89248 ?       Sl   13:22   0:01 /usr/share/elastic-agent/data/elastic-agent-10dc6a/components/heartbeat -E setup.ilm.enabled=false -E setup.template.enabled=false -E management.enabled=true -E management.restart_on_output_change=true -E logging.level=info -E logging.to_stderr=true -E gc_percent=${HEARTBEAT_GOGC:100} -E http.enabled=true -E http.host=unix:///usr/share/elastic-agent/state/data/tmp/synthetics-browser-default.sock -E path.data=/usr/share/elastic-agent/state/data/run/synthetics/browser-default

Capabilities for both processes are the same:

cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap+ep

That env var you asked has the following value: BEAT_SETUID_AS=elastic-agent.

I updated the agent to 8.7.1 but the problem persists.

[emilioalvap (Emilio Alvarez Piñeiro)]

Thanks for the info @tf4, I reviewed the scenario again and it turns out your analysis was right on target.

We have had issues with around mitigations in place to prevent running browser monitors under root. A new approach was implemented on 8.7.0 (not 8.7.1, as I thought) which impacts push monitors and other deprecated types. Since we generally recommend not running containers as root, this issue will impact ECK users mostly.

I've raised an issue here to track.

[DougR]

@emilioalvap

Could you check under what user and capabilities heartbeat process is running? What does the env variable BEAT_SETUID_AS evaluate to inside the container?

This evaluates to elastic-agent.

We have mechanisms in place to prevent running browser monitors as a root user which are probably interfering here. FYI, we have introduced a new approach on 8.7.1.

I just updated to 8.7.1 today. However, in checking my tests, there don't seem to be any changes. I'm still receiving the same results.

We have had issues with around mitigations in place to prevent running browser monitors under root. A new approach was implemented on 8.7.0 (not 8.7.1, as I thought) which impacts push monitors and other deprecated types. Since we generally recommend not running containers as root, this issue will impact ECK users mostly.

A quick question. You write "...push monitors and other deprecated types." Have push monitors been deprecated?

[shahzad31 (Shahzad)]

@DougR Rest of the things @emilioalvap can better answer but we are no way deprecating Push monitors

[emilioalvap (Emilio Alvarez Piñeiro)]

Hi @DougR,

As @shahzad31 very kindly mentioned, push monitors and their related features are not deprecated.

What I meant to say is: other monitor types (mainly zip url and local), which are already deprecated, are also impacted. I should have used better wording on my previous comment, sorry about the confusion.

On the other topics:

• Yes, I initially thought 8.7.1 was the version that came with permissions rework, but it ended up going into 8.7.0.
• elastic-agent is the user that heartbeat will switch to when running synthetic journeys, since chromium can be picky about running as root. ATM, permissions are not granted to this user in order to read journey files from disk.

[DougR]

@emilioalvap

As @shahzad31 very kindly mentioned, push monitors and their related features are not deprecated.

What I meant to say is: other monitor types (mainly zip url and local), which are already deprecated, are also impacted. I should have used better wording on my previous comment, sorry about the confusion.

NP. This is what I assumed, I just wanted clarification, since we've just started to go down this road for our synthetic monitoring.

elastic-agent is the user that heartbeat will switch to when running synthetic journeys, since chromium can be picky about running as root. ATM, permissions are not granted to this user in order to read journey files from disk.

Are there any workarounds for this? I've attempted to set umask for the /tmp directory, as well as attempted to do a setacl + sticky bit on /tmp when the pod starts, but no joy so far.

Thx.

[emilioalvap (Emilio Alvarez Piñeiro)]

Hi @DougR,

Unfortunately, there's no workaround that we can suggest using. Our recommendation would be to upgrade to v8.8 when possible and, in the meantime, running these on our synthetics infrastructure, if that's an option for you.

[DougR]

@emilioalvap

Unfortunately, there's no workaround that we can suggest using. Our recommendation would be to upgrade to v8.8 when possible and, in the meantime, running these on our synthetics infrastructure, if that's an option for you.

Thx. Considering that we're on 8.7.1, is there an ETA for 8.8.x?

[Andrew_Cholakian1 (Andrew Cholakian)]

Unfortunately we can't share expected release dates, it's always possible to find a bug that might cause us to delay a release for instance. It has been a while since our last minor, and it's our goal to get this release out to customers as soon as it's ready. Apologies for the issues you've seen, and we hope 8.8 works much better for you.

[DougR]

I've upgraded to 8.8.0, and everything appears to be working as expected.