# Synthetics failing silently?
**URL:** https://discuss.elastic.co/t/synthetics-failing-silently/295443
**Category:** Synthetics
**Created:** [January 26, 2022, 10:08am UTC](https://discuss.elastic.co/t/synthetics-failing-silently/295443 "2022-01-26T10:08:15Z")
**Posts on this page:** 12
**Page:** 1
### Author:  [@mybyte](https://discuss.elastic.co/u/mybyte)
#### Post date: [January 26, 2022, 10:08am UTC](https://discuss.elastic.co/t/synthetics-failing-silently/295443/1 "2022-01-26T10:08:15Z")
I have a set-up that works perfectly when I run it locally (docker-compose), but is failing without any logs when running on the server.
It's a synthetics test/journey. Configuration is as follows:
mounted /monitors folder in the heartbeat container
The file structure:
```auto
/monitors
someservice.yml
otherservice.yml
syntheticsTest.yml
syntheticsTest/
some.journey.ts
package.json
synthetics.config.ts
```
The config:
```auto
- type: browser
enabled: true
id: someId
name: some name
schedule: "@every 1m"
source.local.path: /monitors/syntheticsTest/
```
The tests run just fine on my local docker stack, so I assume it's something related to the specific environment. I ssh'd into the container and checked all config files and paths - they're fine. So I assume it's something about the environment the container is running in...
Unfortunately, there's no information whatsoever. in the logs:
```auto
2022-01-26T09:54:14.191Z INFO instance/beat.go:492 heartbeat start running.
2022-01-26T09:54:14.191Z INFO beater/heartbeat.go:85 heartbeat is running! Hit CTRL-C to stop it.
2022-01-26T09:54:14.191Z INFO beater/heartbeat.go:87 Effective user/group ids: 1000/1000, with groups: [0]
2022-01-26T09:54:14.191Z INFO beater/heartbeat.go:143 skipping disabled monitor: monitor 'ElasticSearch local' with id 'my-monitor' skipped: monitor not loaded, plugin is disabled
2022-01-26T09:54:14.192Z INFO cfgfile/reload.go:164 Config reloader started
2022-01-26T09:54:19.197Z INFO browser/browser.go:35 Synthetic browser monitor detected! Please note synthetic monitors are a beta feature!
2022-01-26T09:54:19.198Z INFO source/local.go:144 Running /usr/share/heartbeat/.node/node/bin/npm install in /tmp/elastic-synthetics-2179416679
2022-01-26T09:54:20.197Z INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to backoff(elasticsearch(https://elastic:9200))
2022-01-26T09:54:20.198Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2022-01-26T09:54:20.198Z INFO [publisher] pipeline/retry.go:223 done
2022-01-26T09:54:20.198Z WARN [tls] tlscommon/tls_config.go:98 SSL/TLS verifications disabled.
2022-01-26T09:54:20.214Z INFO [esclientleg] eslegclient/connection.go:282 Attempting to connect to Elasticsearch version 7.16.2
2022-01-26T09:54:20.249Z INFO [esclientleg] eslegclient/connection.go:282 Attempting to connect to Elasticsearch version 7.16.2
2022-01-26T09:54:20.266Z INFO [index-management] idxmgmt/std.go:261 Auto ILM enable success.
2022-01-26T09:54:20.267Z INFO [index-management.ilm] ilm/std.go:170 ILM policy heartbeat exists already.
2022-01-26T09:54:20.267Z INFO [index-management] idxmgmt/std.go:397 Set setup.template.name to '{heartbeat-7.16.3 {now/d}-000001}' as ILM is enabled.
2022-01-26T09:54:20.267Z INFO [index-management] idxmgmt/std.go:402 Set setup.template.pattern to 'heartbeat-7.16.3-*' as ILM is enabled.
2022-01-26T09:54:20.267Z INFO [index-management] idxmgmt/std.go:436 Set settings.index.lifecycle.rollover_alias in template to {heartbeat-7.16.3 {now/d}-000001} as ILM is enabled.
2022-01-26T09:54:20.268Z INFO [index-management] idxmgmt/std.go:440 Set settings.index.lifecycle.name in template to {heartbeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2022-01-26T09:54:20.271Z INFO template/load.go:110 Template "heartbeat-7.16.3" already exists and will not be overwritten.
2022-01-26T09:54:20.271Z INFO [index-management] idxmgmt/std.go:297 Loaded index template.
2022-01-26T09:54:20.272Z INFO [index-management.ilm] ilm/std.go:126 Index Alias heartbeat-7.16.3 exists already.
2022-01-26T09:54:20.272Z INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to backoff(elasticsearch(https://elastic:9200)) established
2022-01-26T09:54:44.196Z INFO [monitoring] log/log.go:184 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cgroup":{"cpu":{"cfs"
```
After running npm install there's complete radio silence. So opening an issue in git aside. What would one normally do to narrow down the issue?
---
### Author:  [@BenB196](https://discuss.elastic.co/u/BenB196)
#### Post date: [January 26, 2022, 12:55pm UTC](https://discuss.elastic.co/t/synthetics-failing-silently/295443/2 "2022-01-26T12:55:52Z")
A few questions I'd have are:
1. What version of Heartbeat are you running?
2. Have you tried enabling [debug logging](https://www.elastic.co/guide/en/beats/heartbeat/current/configuration-logging.html#level)? I believe there is some stuff that currently only gets logged at the debug level.
- There is also a way to enable debugging at the playwright level, via the environment variable `DEBUG` being set to `pw:api`.
3. When you say:
> [@mybyte](#):
>
> when running on the server
Is this just a standalone docker server, or something like Kubernetes or Container as a Service platform?
- I ask this because I believe the Heartbeat container depending on version needs to be run as root/privileged container for things to work properly.
---
### Author:  [@mybyte](https://discuss.elastic.co/u/mybyte)
#### Post date: [January 26, 2022, 1:16pm UTC](https://discuss.elastic.co/t/synthetics-failing-silently/295443/3 "2022-01-26T13:16:07Z")
1. 7.16.3
2. which env? Heartbeat container?
3. RHEL podman pod/container with root privileges.
I have a hunch that it's a firewall issue. Took me a while to figure out, but it seems that when running a test suite, synthetics copies the suite to a temp folder, then runs a fresh install before running the suite.
I can ssh into the container and run the test suite manually with `npx @elastic/synthetics .`
We have a few firewall policies and while stuff like npmjs is whitelisted, github is a no-go... for obvious reason. So my guess is that it's trying to pull some additional dependencies or force-refresh libvips (I did a npm install on my machine and copied the node\_modules folder over, didn't help) or chromium and gets stuck at firewall.
Unfortunately, documentation around synthetics is a bit lacking, so I'm not sure how to avoid this.
---
### Author:  [@BenB196](https://discuss.elastic.co/u/BenB196)
#### Post date: [January 26, 2022, 1:48pm UTC](https://discuss.elastic.co/t/synthetics-failing-silently/295443/4 "2022-01-26T13:48:29Z")
> [@mybyte](#):
>
> which env? Heartbeat container?
Yes, adding the environment variable to the Heartbeat container should enable playwright debugging within the container.
> [@mybyte](#):
>
> gets stuck at firewall.
It sounds like you want something similar to an air-gapped solution. I agree that there isn't much around running synthetics in an air-gapped environment. I did however find this [issue](https://github.com/elastic/synthetics/issues/144) which may be able to help.
---
### Author:  [@mybyte](https://discuss.elastic.co/u/mybyte)
#### Post date: [January 26, 2022, 1:51pm UTC](https://discuss.elastic.co/t/synthetics-failing-silently/295443/5 "2022-01-26T13:51:04Z")
Yeah. I've been looking into that and trying to figure out how to get it to work for the last few hours. 馃檪
`docker run [...] -e DEBUG=pw:api docker.elastic.co/beats/heartbeat:7.16.3`
seems to do nothing btw.
---
### Author:  [@lucasfcosta](https://discuss.elastic.co/u/lucasfcosta)
#### Post date: [January 27, 2022, 10:02am UTC](https://discuss.elastic.co/t/synthetics-failing-silently/295443/6 "2022-01-27T10:02:02Z")
Hey @mybyte, thanks for your report. That's really interesting behaviour.
In order for us to narrow down the bug surface of this issue, would you be able to:
a) Confirm whether `which npm` will point to the same path of the `npm` installation Heartbeat is trying to use to run `npm install` (I'm fairly sure it will given it's all within the container, just want to sense-check)
b) If you try to run a different synthetics suite, which has no dependencies, for example one which simply visits `elastic.co`, do you still get stuck at the `npm install` step?
Also just as a note, you mentioned that you also tried to `npm install` within your machine and copy the `node_modules` over. Just one thing to be careful when doing that is to make sure that native dependencies are compiled for the right target. If you're using a MacOS, for example, your execution may fail on an Ubuntu container because some native modules won't work.
---
### Author:  [@mybyte](https://discuss.elastic.co/u/mybyte)
#### Post date: [January 27, 2022, 11:05am UTC](https://discuss.elastic.co/t/synthetics-failing-silently/295443/7 "2022-01-27T11:05:28Z")
Hi @lucasfcosta ,
after spending a day or so, I've narrowed down the issue. I think it's a bug in either synthetics itself or - more likely - the synthetics runner in heartbeat.
I've opened a github issue for this: [Air gapped/offline mode doesn't work 路 Issue #445 路 elastic/synthetics 路 GitHub](https://github.com/elastic/synthetics/issues/445)
There's also a minimal example to reproduce it: [GitHub - mybyte/synthetics-airgapped-issue](https://github.com/mybyte/synthetics-airgapped-issue)
As demonstrated by this example, you could even do this within the heartbeat (7.16.3) container, using its own npm etc. and would still run into issues. It's just the way synthetics seems to be handling external (non-inline) tests. It copies the suite to a temp directory - which might to make sense for clean-up. But then, instead of just running the test suite, it still tries to manipulate the suite by injecting dependencies and such instead of just running it "as is".
It's a shame, really. Because - quite frankly - live-pulling dependencies from [github.com](http://github.com) is a big no-no in an enterprice environment...
---
### Author:  [@lucasfcosta](https://discuss.elastic.co/u/lucasfcosta)
#### Post date: [January 28, 2022, 9:30am UTC](https://discuss.elastic.co/t/synthetics-failing-silently/295443/8 "2022-01-28T09:30:08Z")
Hey @mybyte, thanks a lot for reporting this issue and providing a min. repro. We really appreciate that and will discuss your input when it comes to implementing the NPMless behaviour.
That's currently already on our roadmap here [[Heartbeat] NPM-less default behavior for browser 路 Issue #28749 路 elastic/beats 路 GitHub](https://github.com/elastic/beats/issues/28749), as we don't think the current behaviour is ideal.
However, given the Beta nature of the product, this unfortunately isn't one of our main priorities, so we can't promise when we'll be able to get it.
On another note, Vignesh, from our team, has an example [here](https://github.com/vigneshshanmugam/synthetics/tree/local-packages/examples) of how he managed to use local packages for synthetics checks. It may be worth checking that out to see if you spot any significant differences or still run into any problems.
---
### Author:  [@mybyte](https://discuss.elastic.co/u/mybyte)
#### Post date: [January 28, 2022, 9:50am UTC](https://discuss.elastic.co/t/synthetics-failing-silently/295443/9 "2022-01-28T09:50:15Z")
Yeah, the current approach - even the todo example in the current repo - doesn't work any more. So in current builds (7.16.3), the air gapped example like the one he built won't work any more (see [synthetics/examples/todos at main 路 elastic/synthetics 路 GitHub](https://github.com/elastic/synthetics/tree/main/examples/todos)). Except for /tmp almost all parts of the container all write protected.
Anyhow, due to the bugs / behavior in the script, npm-less behavior won't work.
On a related note:
I found - albeit fairly unclean - workaround for the issue. The problem can be circumvented to some degree by building a container with the npm cache (/usr/share/metricbeat/.npm) baked in. That way, even when running npm install on every single heartbeat run - which honestly should be fixed either way, that's way too resource & time consuming - locally cached resources will be used instead. Got it up and running this way..
---
### Author:  [@Andrew\_Cholakian1](https://discuss.elastic.co/u/Andrew_Cholakian1)
#### Post date: [January 28, 2022, 3:59pm UTC](https://discuss.elastic.co/t/synthetics-failing-silently/295443/10 "2022-01-28T15:59:41Z")
That's a great suggestion re: using locally cached libraries. With the NPM-less behavior ticket that will prevent network requests for anyone only using the synthetics library. The plan is to let people opt-in to supporting additional libraries (since most users just use @elastic/synthetics).
Do you use any NPM modules in addition to `@elastic/synthetics`?
---
### Author:  [@mybyte](https://discuss.elastic.co/u/mybyte)
#### Post date: [January 30, 2022, 3:43pm UTC](https://discuss.elastic.co/t/synthetics-failing-silently/295443/11 "2022-01-30T15:43:21Z")
I didn't. That's the whole deal.
As far as I can tell, between the time the current container (7.16.3) of heartbeat was built and now, some parts of the dependency chain have changed. NPM was trying to access github and download the latest version of [GitHub - lovell/sharp-libvips: Packaging scripts to prebuild libvips and its dependencies - you're probably looking for https://github.com/lovell/sharp](https://github.com/lovell/sharp-libvips).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 23, 2022, 3:43pm UTC](https://discuss.elastic.co/t/synthetics-failing-silently/295443/12 "2022-02-23T15:43:37Z")
This topic was automatically closed 24 days after the last reply. New replies are no longer allowed.