# Elastic Synthetics Journey: Receiving \`permission denied\`
**URL:** https://discuss.elastic.co/t/elastic-synthetics-journey-receiving-permission-denied/330032
**Category:** Synthetics
**Tags:** docker
**Created:** [April 14, 2023, 5:05pm UTC](https://discuss.elastic.co/t/elastic-synthetics-journey-receiving-permission-denied/330032 "2023-04-14T17:05:31Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@DougR](https://discuss.elastic.co/u/DougR)
#### Post date: [April 14, 2023, 5:05pm UTC](https://discuss.elastic.co/t/elastic-synthetics-journey-receiving-permission-denied/330032/1 "2023-04-14T17:05:31Z")
I am deploying a journey to Elastic Synthetics using the following command: `npm run push`.
I'm using Elastic Cloud 8.7.0 and my agent is an `elastic-agent-complete:8.7.0` container.
The test is pushing to Elastic Cloud as expected, however it does not run and stays in pending status. When I run the test locally using `npx @elastic/synthetics .`, it runs as expected and succeeds.
When I check the agent logs for the test ID, these are the log entries I find that appear to be relevant:
```auto
{"log.level":"info","@timestamp":"2023-04-14T15:09:48.602Z","message":"Running /usr/share/elastic-agent/.node/node/bin/npm install in /tmp/elastic-synthetics-unzip-3786772281","component":{"binary":"heartbeat","dataset":"elastic_agent.heartbeat","id":"synthetics/browser-default","type":"synthetics/browser"},"log":{"source":"synthetics/browser-default"},"log.origin":{"file.line":148,"file.name":"source/local.go"},"service.name":"heartbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-14T15:09:51.429Z","message":"Running command: /tmp/elastic-synthetics-unzip-3786772281/node_modules/.bin/elastic-synthetics /tmp/elastic-synthetics-unzip-3786772281/node_modules/.bin/elastic-synthetics /tmp/elastic-synthetics-unzip-3786772281 --playwright-options {\"headless\":true,\"ignoreHTTPSErrors\":false} --screenshots on --throttling 5d/3u/20l --rich-events --match MyApp Dev --params \"{2 hidden params}\" in directory: '/tmp/elastic-synthetics-unzip-3786772281'","component":{"binary":"heartbeat","dataset":"elastic_agent.heartbeat","id":"synthetics/browser-default","type":"synthetics/browser"},"log":{"source":"synthetics/browser-default"},"service.name":"heartbeat","ecs.version":"1.6.0","log.origin":{"file.line":170,"file.name":"synthexec/synthexec.go"},"ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-04-14T15:09:51.429Z","message":"Could not start command /tmp/elastic-synthetics-unzip-3786772281/node_modules/.bin/elastic-synthetics /tmp/elastic-synthetics-unzip-3786772281 --playwright-options {\"headless\":true,\"ignoreHTTPSErrors\":false} --screenshots on --throttling 5d/3u/20l --rich-events --match MyApp Dev --params {\"dev\":{\"id\":\"179aa0f6-7d83-4980-aacf-ee932e7df83b\",\"journey\":\"MyApp Dev\",\"url\":\"https://myapp-dev.example.com/\",\"validation\":\"MyCompany\"},\"prod\":{\"id\":\"c95acd45-b81b-4473-8f6a-fa98f3b175cc\",\"journey\":\"MyApp Prod\",\"url\":\"https://myapp-dev.example.com/\",\"validation\":\"MyCompany\"}} --outfd 3: fork/exec /tmp/elastic-synthetics-unzip-3786772281/node_modules/.bin/elastic-synthetics: permission denied","component":{"binary":"heartbeat","dataset":"elastic_agent.heartbeat","id":"synthetics/browser-default","type":"synthetics/browser"},"log":{"source":"synthetics/browser-default"},"log.origin":{"file.line":251,"file.name":"synthexec/synthexec.go"},"service.name":"heartbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
```
`/tmp/elastic-synthetics-unzip-3786772281/node_modules/.bin/` contains the following:
```auto
lrwxrwxrwx 1 root root 34 Apr 14 15:09 elastic-synthetics -> ../@elastic/synthetics/dist/cli.js
lrwxrwxrwx 1 root root 34 Apr 14 15:09 synthetics -> ../@elastic/synthetics/dist/cli.js
```
This resolves to the directory `/tmp/elastic-synthetics-unzip-3786772281/node_modules/@elastic/synthetics/dist`, which contains:
```auto
-rwxrwx--- 1 elastic-agent elastic-agent 8479 Mar 28 04:07 cli.js*
```
This has 0770 perms which should be executable for user/group and contains the following shebang:
```auto
#!/usr/bin/env node
```
...so I'm uncertain what is causing the permission denied.
---
### Author:  [@spaulovich](https://discuss.elastic.co/u/spaulovich)
#### Post date: [April 17, 2023, 5:49pm UTC](https://discuss.elastic.co/t/elastic-synthetics-journey-receiving-permission-denied/330032/2 "2023-04-17T17:49:07Z")
Experiencing the same issue.
Stuck in pending state with agent log showing "Could not start command /tmp/elastic-synthetics-unzip... : permission denied"
---
### Author:  [@Andrew\_Cholakian1](https://discuss.elastic.co/u/Andrew_Cholakian1)
#### Post date: [April 17, 2023, 6:57pm UTC](https://discuss.elastic.co/t/elastic-synthetics-journey-receiving-permission-denied/330032/3 "2023-04-17T18:57:39Z")
It sounds like you're using zip monitors which have been deprecated for some time now in favor of the more powerful and featureful [project monitors](https://www.elastic.co/guide/en/observability/current/synthetics-get-started-project.html). Have you tried those? They should operate more smoothly and make deployment easier.
---
### Author:  [@spaulovich](https://discuss.elastic.co/u/spaulovich)
#### Post date: [April 17, 2023, 7:34pm UTC](https://discuss.elastic.co/t/elastic-synthetics-journey-receiving-permission-denied/330032/4 "2023-04-17T19:34:23Z")
I don't understand. These ARE project monitors, built in a repository, pushed to our Kibana server, and attempting to run in a private location.
As a prototype, we created a browser-based monitor through Kibana UI. It's been running in that private location without incident. But when we push tests in a project monitor repo, they encounter this permission error.
---
### Author:  [@Andrew\_Cholakian1](https://discuss.elastic.co/u/Andrew_Cholakian1)
#### Post date: [April 17, 2023, 8:02pm UTC](https://discuss.elastic.co/t/elastic-synthetics-journey-receiving-permission-denied/330032/5 "2023-04-17T20:02:06Z")
I apologize, I misread the logfile.
I suspect there's something special about how the container is being launched, I currently have a private location running with that same exact version of the `elastic-agent-complete` image successfully.
How are you running the container? Are you running it under a special user or mounting any volumes? Is there a customized seccomp policy? I'm looking for anything out of the ordinary that could cause those perms errors.
---
### Author:  [@spaulovich](https://discuss.elastic.co/u/spaulovich)
#### Post date: [April 17, 2023, 8:17pm UTC](https://discuss.elastic.co/t/elastic-synthetics-journey-receiving-permission-denied/330032/6 "2023-04-17T20:17:37Z")
This is our agent manifest:
apiVersion: [agent.k8s.elastic.co/v1alpha1](http://agent.k8s.elastic.co/v1alpha1)
kind: Agent
metadata:
name: elastic-agent
namespace: elasticsearch
spec:
version: 8.7.0
image: [docker.elastic.co/beats/elastic-agent-complete:8.7.0](http://docker.elastic.co/beats/elastic-agent-complete:8.7.0)
kibanaRef:
name: kibana
fleetServerRef:
name: fleet-server
mode: fleet
deployment:
replicas: 3
podTemplate:
spec:
securityContext:
runAsUser: 0
capabilities:
add: ["NET\_RAW", "SETUID"]
containers:
- name: agent
resources:
limits:
memory: 4Gi
cpu: 3
requests:
memory: 4Gi
cpu: 3
---
### Author:  [@DougR](https://discuss.elastic.co/u/DougR)
#### Post date: [April 18, 2023, 2:15pm UTC](https://discuss.elastic.co/t/elastic-synthetics-journey-receiving-permission-denied/330032/7 "2023-04-18T14:15:47Z")
> [@Andrew\_Cholakian1](#):
>
> It sounds like you're using zip monitors which have been deprecated for some time now in favor of the more powerful and featureful [project monitors](https://www.elastic.co/guide/en/observability/current/synthetics-get-started-project.html). Have you tried those? They should operate more smoothly and make deployment easier.
I'm not using zip monitors. I'm using project monitors developed with the Node.js `@elastic/synthetics` library from a project initialized with `npx @elastic/synthetics init .`
---
### Author:  [@DougR](https://discuss.elastic.co/u/DougR)
#### Post date: [April 18, 2023, 2:29pm UTC](https://discuss.elastic.co/t/elastic-synthetics-journey-receiving-permission-denied/330032/8 "2023-04-18T14:29:52Z")
> [@Andrew\_Cholakian1](#):
>
> I apologize, I misread the logfile.
NP
> I suspect there's something special about how the container is being launched, I currently have a private location running with that same exact version of the `elastic-agent-complete` image successfully.
Just to confirm: Scripted browser tests deployed from a project using `npm run push` are functioning as expected, correct?
> How are you running the container? Are you running it under a special user or mounting any volumes? Is there a customized seccomp policy? I'm looking for anything out of the ordinary that could cause those perms errors.
I'm launching using the Elastic Operator. I do have a few additional volumes mounted, in order to add self-signed certificates to the pod at startup, using a customized version of `docker-entrypoint` that also fixes the issue with `elastic-agent` that occasionally leaves lockfiles in place when a pod stops.
My Kubernetes manifest is:
```yaml
---
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
name: elastic-synthetics
spec:
deployment:
podTemplate:
metadata:
annotations:
sidecar.istio.io/inject: "false"
labels:
env: dev
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchExpressions:
- key: agent.k8s.elastic.co/name
operator: In
values:
- elastic-synthetics
topologyKey: kubernetes.io/hostname
weight: 100
containers:
- env:
- name: CONFIG_PATH
value: /usr/share/elastic-agent
- name: ELASTIC_AGENT_TAGS
valueFrom:
fieldRef:
fieldPath: metadata.labels['env']
- name: ENV
valueFrom:
fieldRef:
fieldPath: metadata.labels['env']
- name: HOST_IP
valueFrom:
fieldRef:
fieldPath: status.hostIP
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
envFrom:
- configMapRef:
name: elastic-synthetics-env
- secretRef:
name: elastic-synthetics-secret
name: agent
resources:
limits:
cpu: 1000m
memory: 2Gi
requests:
cpu: 1000m
memory: 2Gi
securityContext:
runAsUser: 0
volumeMounts:
- mountPath: /usr/local/share/ca-certificates
name: certs
readOnly: true
- mountPath: /usr/local/bin
name: config
readOnly: true
- mountPath: /tmp/nssdb
name: nssdb
readOnly: true
priorityClassName: system-cluster-critical
securityContext:
runAsUser: 0
volumes:
- name: certs
secret:
secretName: athene-certs-secret
- configMap:
defaultMode: 493
name: elastic-synthetics-config
name: config
- name: nssdb
secret:
defaultMode: 438
secretName: elastic-synthetics-nssdb-secret
replicas: 1
strategy:
rollingUpdate:
maxUnavailable: 10%
type: RollingUpdate
mode: fleet
version: 8.7.0
```
# Edit
This is a simple browser test that I'm using and which is stuck in pending. It succeeds when run locally.
```typescript
import { journey, step, monitor, expect } from '@elastic/synthetics';
journey('Google Test', ({ page, params }) => {
monitor.use({
id: '2252824d-ba39-454e-8ebd-d749de23279f',
schedule: 10,
});
step('Open https://google.com', async () => {
await page.goto('https://google.com');
await expect(page.getByText('Google')).toBeTruthy();
});
});
```
---
### Author:  [@DougR](https://discuss.elastic.co/u/DougR)
#### Post date: [April 18, 2023, 2:48pm UTC](https://discuss.elastic.co/t/elastic-synthetics-journey-receiving-permission-denied/330032/9 "2023-04-18T14:48:01Z")
> [@Andrew\_Cholakian1](#):
>
> I suspect there's something special about how the container is being launched, I currently have a private location running with that same exact version of the `elastic-agent-complete` image successfully.
>
> How are you running the container? Are you running it under a special user or mounting any volumes? Is there a customized seccomp policy? I'm looking for anything out of the ordinary that could cause those perms errors.
I vaguely remember when I first read that I needed to use `elastic-agent-complete` that there were specific environment variables that needed to be set for synthetics to work properly. However, I haven't been able to find that reference again. Am I remembering correctly? If so, what are these variables and what should the values be so that I can confirm that I have them set?
---
### Author:  [@DougR](https://discuss.elastic.co/u/DougR)
#### Post date: [April 18, 2023, 6:05pm UTC](https://discuss.elastic.co/t/elastic-synthetics-journey-receiving-permission-denied/330032/10 "2023-04-18T18:05:15Z")
> [@DougR](#):
>
> This is a simple browser test that I'm using and which is stuck in pending. It succeeds when run locally.
>
> ```auto
> import { journey, step, monitor, expect } from '@elastic/synthetics';
>
> journey('Google Test', ({ page, params }) => {
> monitor.use({
> id: '2252824d-ba39-454e-8ebd-d749de23279f',
> schedule: 10,
> });
>
> step('Open https://google.com', async () => {
> await page.goto('https://google.com');
> await expect(page.getByText('Google')).toBeTruthy();
> });
> });
>
> ```
# Updates
## Update #1
When I create a multi-step journey in the UI and paste this script in, it fails, but it does attempt to run.
## Update #2
I figured out what the issue was and it now runs as expected when I create a multi-step journey with the above script in the UI. Still not working when pushed from CLI.
---
### Author:  [@DougR](https://discuss.elastic.co/u/DougR)
#### Post date: [April 19, 2023, 2:11pm UTC](https://discuss.elastic.co/t/elastic-synthetics-journey-receiving-permission-denied/330032/11 "2023-04-19T14:11:22Z")
I found the following in the [v8.7 documentation](https://www.elastic.co/guide/en/fleet/8.7/elastic-agent-container.html#_step_3_run_the_elastic_agent_image):
> Synthetic tests cannot run under the root user. Refer to [Synthetics Fleet Quickstart](https://www.elastic.co/guide/en/observability/8.7/uptime-set-up.html) for more information.
I have to wonder whether this is the issue. When I look at the processes running in the container, they're all running as `root`. Are all synthetics test pushed through the UI automagically run as `elastic-agent`? OTOH, if this were the issue, I would expect the lightweight tests I'm pushing via `@elastic/synthetics` to also fail.
@Andrew_Cholakian1 - Since your deployment is working as expected, can you provide any insight?
---
### Author:  [@DougR](https://discuss.elastic.co/u/DougR)
#### Post date: [April 21, 2023, 6:21pm UTC](https://discuss.elastic.co/t/elastic-synthetics-journey-receiving-permission-denied/330032/12 "2023-04-21T18:21:05Z")
When I attempt to deploy a browser test to an on-prem node (e.g., an Elastic Agent deployed to a Linux server, rather than to an `elastic-agent-complete` container), I get the following error:
---
### Author:  [@tf4](https://discuss.elastic.co/u/tf4)
#### Post date: [April 27, 2023, 5:49pm UTC](https://discuss.elastic.co/t/elastic-synthetics-journey-receiving-permission-denied/330032/13 "2023-04-27T17:49:12Z")
I'm also stuck into the same error. Browser monitors run correctly when created through Kibana but they don't when created using project monitors.
I tried changing the user of my k8s container to "elastic-agent" as described in the docs but it gives me other permission errors while starting up the container.
---
### Author:  [@DougR](https://discuss.elastic.co/u/DougR)
#### Post date: [April 27, 2023, 7:03pm UTC](https://discuss.elastic.co/t/elastic-synthetics-journey-receiving-permission-denied/330032/14 "2023-04-27T19:03:01Z")
> [@tf4](#):
>
> I tried changing the user of my k8s container to "elastic-agent" as described in the docs but it gives me other permission errors while starting up the container.
Same here.
---
### Author:  [@Andrew\_Cholakian1](https://discuss.elastic.co/u/Andrew_Cholakian1)
#### Post date: [April 27, 2023, 7:59pm UTC](https://discuss.elastic.co/t/elastic-synthetics-journey-receiving-permission-denied/330032/15 "2023-04-27T19:59:50Z")
Sorry for the delay here, trying to balance a number of priorities.
@tf4 what are the errors you're getting?
We're probably a week or so out from being able to dedicate some time to look into this unfortunately.
---
### Author:  [@tf4](https://discuss.elastic.co/u/tf4)
#### Post date: [April 27, 2023, 9:09pm UTC](https://discuss.elastic.co/t/elastic-synthetics-journey-receiving-permission-denied/330032/16 "2023-04-27T21:09:18Z")
@Andrew_Cholakian1 I'm getting exactly the same error that @DougR mentioned in the beginning of this thread. BTW I'm using ECK and `elastic-agent-complete:8.7.0` as well.
When I change the user to `elastic-agent` (`runAsUser: 1000`) I keep getting the following error and the container doesn't start.
```auto
cp: cannot create regular file '/usr/local/share/ca-certificates/ca.crt': Permission denied
```
I know this is related to the default volume mounts of the agent but I believe I shouldn't change anything in those volumes.
---
### Author:  [@tf4](https://discuss.elastic.co/u/tf4)
#### Post date: [April 27, 2023, 9:21pm UTC](https://discuss.elastic.co/t/elastic-synthetics-journey-receiving-permission-denied/330032/17 "2023-04-27T21:21:45Z")
In my last investigation I checked the permissions of the folders inside `/tmp` and discovered that my monitors are being copied to the agent with permissions to the root user only. I suspect this is the reason why the `elastic-agent` user can't execute that `.bin/elastic-synthetics` command.
---
### Author:  [@emilioalvap](https://discuss.elastic.co/u/emilioalvap)
#### Post date: [May 4, 2023, 9:02am UTC](https://discuss.elastic.co/t/elastic-synthetics-journey-receiving-permission-denied/330032/18 "2023-05-04T09:02:37Z")
Hi @DougR, @tf4,
Elastic agent cannot run as non-root user, it's a [known issue for ECK deployments](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-elastic-agent-fleet-known-limitations.html#k8s_running_as_root_and_within_a_single_namespace). Any errors you get when changing the user are probably related to that.
Could you check under what user and capabilities `heartbeat` process is running? What does the env variable `BEAT_SETUID_AS` evaluate to inside the container?
We have mechanisms in place to prevent running browser monitors as a root user which are probably interfering here. FYI, we have introduced a new approach on `8.7.1`.
---
### Author:  [@tf4](https://discuss.elastic.co/u/tf4)
#### Post date: [May 4, 2023, 1:46pm UTC](https://discuss.elastic.co/t/elastic-synthetics-journey-receiving-permission-denied/330032/19 "2023-05-04T13:46:46Z")
Hi @emilioalvap!
Thanks for the info about running as non-root user.
As you can see in the output below, there are two heartbeat processes running as `root`.
```auto
root 924 0.1 0.5 1503900 88436 ? Sl 13:22 0:01 /usr/share/elastic-agent/data/elastic-agent-10dc6a/components/heartbeat -E setup.ilm.enabled=false -E setup.template.enabled=false -E management.enabled=true -E management.restart_on_output_change=true -E logging.level=info -E logging.to_stderr=true -E gc_percent=${HEARTBEAT_GOGC:100} -E http.enabled=true -E http.host=unix:///usr/share/elastic-agent/state/data/tmp/synthetics-http-default.sock -E path.data=/usr/share/elastic-agent/state/data/run/synthetics/http-default
root 942 0.2 0.5 2565076 89248 ? Sl 13:22 0:01 /usr/share/elastic-agent/data/elastic-agent-10dc6a/components/heartbeat -E setup.ilm.enabled=false -E setup.template.enabled=false -E management.enabled=true -E management.restart_on_output_change=true -E logging.level=info -E logging.to_stderr=true -E gc_percent=${HEARTBEAT_GOGC:100} -E http.enabled=true -E http.host=unix:///usr/share/elastic-agent/state/data/tmp/synthetics-browser-default.sock -E path.data=/usr/share/elastic-agent/state/data/run/synthetics/browser-default
```
Capabilities for both processes are the same:
```auto
cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap+ep
```
That env var you asked has the following value: `BEAT_SETUID_AS=elastic-agent`.
I updated the agent to `8.7.1` but the problem persists.
---
### Author:  [@emilioalvap](https://discuss.elastic.co/u/emilioalvap)
#### Post date: [May 5, 2023, 10:16am UTC](https://discuss.elastic.co/t/elastic-synthetics-journey-receiving-permission-denied/330032/20 "2023-05-05T10:16:31Z")
Thanks for the info @tf4, I reviewed the scenario again and it turns out [your analysis](https://discuss.elastic.co/t/elastic-synthetics-journey-receiving-permission-denied/330032/17) was right on target.
We have had issues with around mitigations in place to prevent running browser monitors under root. A new approach was implemented on 8.7.0 (not 8.7.1, as I thought) which impacts `push` monitors and other deprecated types. Since we generally recommend not running containers as root, this issue will impact ECK users mostly.
I've raised an issue [here](https://github.com/elastic/beats/issues/35343) to track.
[Next page](https://discuss.elastic.co/t/elastic-synthetics-journey-receiving-permission-denied/330032.md?page=2)