Need to disable all the steps snaps in Elastic Synthetic

shahidraza · April 12, 2021, 4:53pm

Hi Team,

Are there any properties that allow us to capture images only when any errors happen in the configured flows? As of now, we can see more than 1 TB of Data utilized within 1 month and only with 2-3 flows.

I am ok if we don't capture images as well.

Currently, I am using 7.11 Synthetic packages.

Thanks in advance.

shahidraza · April 13, 2021, 9:48am

@Andrew_Cholakian1 Please suggest if any possibilities.

shahzad31 · April 19, 2021, 3:59am

Hi @shahidraza , on synthetics we do support the option CLI Options of setings
--screenshots=false
to not capture screenshots, depending on your setup you can pass that flag. In that case it will not capture step snaps.

Regards

drewpost · April 19, 2021, 12:17pm

Hi @shahidraza - Just to expand on Shahzad's reply, we're looking at putting in proper lifecycle management and scenario rules on when to capture screenshots as well as how long to keep the really rich object-level detail. Whilst his solution is a good temporary fix to the significant data volumes you're seeing, I just wanted to let you know that we will be addressing this properly as well.

shahidraza · April 21, 2021, 9:47am

@shahzad31 what is the flag I can use in heartbeat.docker.yml.

I am using heartbeat.docker.yml and todo folder to run my docker synthetic environment.

Thanks for sharing the option --screenshots=false but please share the parameters which I can use in my configuration either in heartbeat.docker.yml or run.sh file.

Thanks in advance.

Regards,
Shahid

shahidraza · April 21, 2021, 9:47am

Awesome news @drewpost

shahidraza · April 21, 2021, 10:00am

@shahzad31 I made below changes in parse_args.ts file but its not working. Please suggest

option('--screenshots=false', 'take screenshot for each step')
option('--screenshots', 'false')

Andrew_Cholakian1 · April 21, 2021, 12:49pm

To be clear, this needs to be present in Heartbeat as an option to be passed through, apologies for the confusion here. I'm thinking we should prioritize switching to jpeg screenshots here: Add option for JPEG screenshots · Issue #233 · elastic/synthetics · GitHub as a first step, that should cut down on file size to start with.

I've opened [Heartbeat] Flexible screenshot functionality · Issue #25201 · elastic/beats · GitHub to track the heartbeat side of this.

In the meantime, you could always use Elasticsearch's delete by query feature to remove just screenshots, here's an example:

gist.github.com

https://gist.github.com/andrewvc/b9361cd080ac6931173869ff61103f3f

reducing-disk-space.json

# WARNING: This will delete all heartbeat network info and screenshot docs older than 7 days ago at midnight
# For large datasets its a good idea to set wait_for_completion to false since the request
# may block for a while
POST heartbeat-*/_delete_by_query?wait_for_completion=false
{
  "query": {
    "bool": {
      "filter": [
        {
          "range": {

This file has been truncated. show original

shahidraza · April 23, 2021, 3:19pm

@vigneshshanmugam Hey do we have option to disable the every steps snapshot?

Thanks in advance

Regards,
Shahid

vigneshshanmugam · April 26, 2021, 2:58pm

Hi @shahidraza,

The new version of Synthetics library 1.0.0-beta.0 already uses the JPEG images for step screenshots, which should greatly reduce the storage as we saw 50% reduction in the image size for some of our tests. Please update your suite to point to that version - synthetics/package.json at 86ca02d47b6eab8d1b7e88385b66123b0a36a164 · elastic/synthetics · GitHub

But our long term strategy that would work for everyone is to prioritize this issue - [Heartbeat] Flexible screenshot functionality · Issue #25201 · elastic/beats · GitHub as Andrew mentioned and provide a way to configure heartbeat.

Let us know if you need any other help.

Thanks,
Vignesh

system · May 20, 2021, 2:58pm

This topic was automatically closed 24 days after the last reply. New replies are no longer allowed.