Synthetics failing silently?

mybyte · January 26, 2022, 10:08am

I have a set-up that works perfectly when I run it locally (docker-compose), but is failing without any logs when running on the server.

It's a synthetics test/journey. Configuration is as follows:

mounted /monitors folder in the heartbeat container
The file structure:

/monitors
   someservice.yml
   otherservice.yml
   syntheticsTest.yml
   syntheticsTest/
      some.journey.ts
      package.json
      synthetics.config.ts

The config:

- type: browser
  enabled: true
  id: someId
  name: some name
  schedule: "@every 1m"
  source.local.path: /monitors/syntheticsTest/

The tests run just fine on my local docker stack, so I assume it's something related to the specific environment. I ssh'd into the container and checked all config files and paths - they're fine. So I assume it's something about the environment the container is running in...

Unfortunately, there's no information whatsoever. in the logs:

2022-01-26T09:54:14.191Z        INFO    instance/beat.go:492    heartbeat start running.
2022-01-26T09:54:14.191Z        INFO    beater/heartbeat.go:85  heartbeat is running! Hit CTRL-C to stop it.
2022-01-26T09:54:14.191Z        INFO    beater/heartbeat.go:87  Effective user/group ids: 1000/1000, with groups: [0]
2022-01-26T09:54:14.191Z        INFO    beater/heartbeat.go:143 skipping disabled monitor: monitor 'ElasticSearch local' with id 'my-monitor' skipped: monitor not loaded, plugin is disabled
2022-01-26T09:54:14.192Z        INFO    cfgfile/reload.go:164   Config reloader started
2022-01-26T09:54:19.197Z        INFO    browser/browser.go:35   Synthetic browser monitor detected! Please note synthetic monitors are a beta feature!
2022-01-26T09:54:19.198Z        INFO    source/local.go:144     Running /usr/share/heartbeat/.node/node/bin/npm install in /tmp/elastic-synthetics-2179416679
2022-01-26T09:54:20.197Z        INFO    [publisher_pipeline_output]     pipeline/output.go:143  Connecting to backoff(elasticsearch(https://elastic:9200))
2022-01-26T09:54:20.198Z        INFO    [publisher]     pipeline/retry.go:219   retryer: send unwait signal to consumer
2022-01-26T09:54:20.198Z        INFO    [publisher]     pipeline/retry.go:223     done
2022-01-26T09:54:20.198Z        WARN    [tls]   tlscommon/tls_config.go:98      SSL/TLS verifications disabled.
2022-01-26T09:54:20.214Z        INFO    [esclientleg]   eslegclient/connection.go:282   Attempting to connect to Elasticsearch version 7.16.2
2022-01-26T09:54:20.249Z        INFO    [esclientleg]   eslegclient/connection.go:282   Attempting to connect to Elasticsearch version 7.16.2
2022-01-26T09:54:20.266Z        INFO    [index-management]      idxmgmt/std.go:261      Auto ILM enable success.
2022-01-26T09:54:20.267Z        INFO    [index-management.ilm]  ilm/std.go:170  ILM policy heartbeat exists already.
2022-01-26T09:54:20.267Z        INFO    [index-management]      idxmgmt/std.go:397      Set setup.template.name to '{heartbeat-7.16.3 {now/d}-000001}' as ILM is enabled.
2022-01-26T09:54:20.267Z        INFO    [index-management]      idxmgmt/std.go:402      Set setup.template.pattern to 'heartbeat-7.16.3-*' as ILM is enabled.
2022-01-26T09:54:20.267Z        INFO    [index-management]      idxmgmt/std.go:436      Set settings.index.lifecycle.rollover_alias in template to {heartbeat-7.16.3 {now/d}-000001} as ILM is enabled.
2022-01-26T09:54:20.268Z        INFO    [index-management]      idxmgmt/std.go:440      Set settings.index.lifecycle.name in template to {heartbeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2022-01-26T09:54:20.271Z        INFO    template/load.go:110    Template "heartbeat-7.16.3" already exists and will not be overwritten.
2022-01-26T09:54:20.271Z        INFO    [index-management]      idxmgmt/std.go:297      Loaded index template.
2022-01-26T09:54:20.272Z        INFO    [index-management.ilm]  ilm/std.go:126  Index Alias heartbeat-7.16.3 exists already.
2022-01-26T09:54:20.272Z        INFO    [publisher_pipeline_output]     pipeline/output.go:151  Connection to backoff(elasticsearch(https://elastic:9200)) established
2022-01-26T09:54:44.196Z        INFO    [monitoring]    log/log.go:184  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cgroup":{"cpu":{"cfs"

After running npm install there's complete radio silence. So opening an issue in git aside. What would one normally do to narrow down the issue?

BenB196 · January 26, 2022, 12:55pm

A few questions I'd have are:

What version of Heartbeat are you running?
Have you tried enabling debug logging? I believe there is some stuff that currently only gets logged at the debug level.
- There is also a way to enable debugging at the playwright level, via the environment variable DEBUG being set to pw:api.
When you say:

Is this just a standalone docker server, or something like Kubernetes or Container as a Service platform?

I ask this because I believe the Heartbeat container depending on version needs to be run as root/privileged container for things to work properly.

mybyte · January 26, 2022, 1:16pm

7.16.3
which env? Heartbeat container?
RHEL podman pod/container with root privileges.

I have a hunch that it's a firewall issue. Took me a while to figure out, but it seems that when running a test suite, synthetics copies the suite to a temp folder, then runs a fresh install before running the suite.

I can ssh into the container and run the test suite manually with npx @elastic/synthetics .
We have a few firewall policies and while stuff like npmjs is whitelisted, github is a no-go... for obvious reason. So my guess is that it's trying to pull some additional dependencies or force-refresh libvips (I did a npm install on my machine and copied the node_modules folder over, didn't help) or chromium and gets stuck at firewall.

Unfortunately, documentation around synthetics is a bit lacking, so I'm not sure how to avoid this.

BenB196 · January 26, 2022, 1:48pm

Yes, adding the environment variable to the Heartbeat container should enable playwright debugging within the container.

It sounds like you want something similar to an air-gapped solution. I agree that there isn't much around running synthetics in an air-gapped environment. I did however find this issue which may be able to help.

mybyte · January 26, 2022, 1:51pm

Yeah. I've been looking into that and trying to figure out how to get it to work for the last few hours.
docker run [...] -e DEBUG=pw:api docker.elastic.co/beats/heartbeat:7.16.3
seems to do nothing btw.

lucasfcosta · January 27, 2022, 10:02am

Hey @mybyte, thanks for your report. That's really interesting behaviour.

In order for us to narrow down the bug surface of this issue, would you be able to:

a) Confirm whether which npm will point to the same path of the npm installation Heartbeat is trying to use to run npm install (I'm fairly sure it will given it's all within the container, just want to sense-check)
b) If you try to run a different synthetics suite, which has no dependencies, for example one which simply visits elastic.co, do you still get stuck at the npm install step?

Also just as a note, you mentioned that you also tried to npm install within your machine and copy the node_modules over. Just one thing to be careful when doing that is to make sure that native dependencies are compiled for the right target. If you're using a MacOS, for example, your execution may fail on an Ubuntu container because some native modules won't work.

mybyte · January 27, 2022, 11:05am

Hi @lucasfcosta ,

after spending a day or so, I've narrowed down the issue. I think it's a bug in either synthetics itself or - more likely - the synthetics runner in heartbeat.

I've opened a github issue for this: Air gapped/offline mode doesn't work · Issue #445 · elastic/synthetics · GitHub
There's also a minimal example to reproduce it: GitHub - mybyte/synthetics-airgapped-issue

As demonstrated by this example, you could even do this within the heartbeat (7.16.3) container, using its own npm etc. and would still run into issues. It's just the way synthetics seems to be handling external (non-inline) tests. It copies the suite to a temp directory - which might to make sense for clean-up. But then, instead of just running the test suite, it still tries to manipulate the suite by injecting dependencies and such instead of just running it "as is".

It's a shame, really. Because - quite frankly - live-pulling dependencies from github.com is a big no-no in an enterprice environment...

lucasfcosta · January 28, 2022, 9:30am

Hey @mybyte, thanks a lot for reporting this issue and providing a min. repro. We really appreciate that and will discuss your input when it comes to implementing the NPMless behaviour.

That's currently already on our roadmap here [Heartbeat] NPM-less default behavior for browser · Issue #28749 · elastic/beats · GitHub, as we don't think the current behaviour is ideal.

However, given the Beta nature of the product, this unfortunately isn't one of our main priorities, so we can't promise when we'll be able to get it.

On another note, Vignesh, from our team, has an example here of how he managed to use local packages for synthetics checks. It may be worth checking that out to see if you spot any significant differences or still run into any problems.

mybyte · January 28, 2022, 9:50am

Yeah, the current approach - even the todo example in the current repo - doesn't work any more. So in current builds (7.16.3), the air gapped example like the one he built won't work any more (see synthetics/examples/todos at main · elastic/synthetics · GitHub). Except for /tmp almost all parts of the container all write protected.

Anyhow, due to the bugs / behavior in the script, npm-less behavior won't work.

On a related note:
I found - albeit fairly unclean - workaround for the issue. The problem can be circumvented to some degree by building a container with the npm cache (/usr/share/metricbeat/.npm) baked in. That way, even when running npm install on every single heartbeat run - which honestly should be fixed either way, that's way too resource & time consuming - locally cached resources will be used instead. Got it up and running this way..

Andrew_Cholakian1 · January 28, 2022, 3:59pm

That's a great suggestion re: using locally cached libraries. With the NPM-less behavior ticket that will prevent network requests for anyone only using the synthetics library. The plan is to let people opt-in to supporting additional libraries (since most users just use @elastic/synthetics).

Do you use any NPM modules in addition to @elastic/synthetics?

mybyte · January 30, 2022, 3:43pm

I didn't. That's the whole deal.

As far as I can tell, between the time the current container (7.16.3) of heartbeat was built and now, some parts of the dependency chain have changed. NPM was trying to access github and download the latest version of GitHub - lovell/sharp-libvips: Packaging scripts to prebuild libvips and its dependencies - you're probably looking for https://github.com/lovell/sharp.

system · February 23, 2022, 3:43pm

This topic was automatically closed 24 days after the last reply. New replies are no longer allowed.