Are there any plans to increase the 800kb push limit?

Elastic Observability Synthetics
1

I want to run a set of tests that will check the contents of PDF files using the pdf-parse node module. However, when I import it to use in my monitors I can't push it up to elastic as I get the following error:
Error: You have monitors whose size exceeds the 800KB limit.

Are there any plans to increase this size limit?

2

Hi @joel.parker,

Thanks for reporting. We didnt have any plan to increase the size limit as of now, the main restriction was to catch any major JS bundles that gets in to the journey script without users notice. But we just checked the size of the pdf-parse module and its well within the size restirction.

pdf-parse v1.1.1 ❘ Bundlephobia - Reports it as ~300Kb. Are you trying to push more than x amount of monitors at same time?

Thanks,
Vignesh

3

Thanks, I am only pushing one monitor but I have had to directly import the file 'pdf-parse/lib/pdf.js/v1.10.100/build/pdf' because when running in Elastic I am receiving an error that this file cannot be found. I'm not sure if it's something to do with how the node_modules folder is zipped/unzipped when running via Elastic?

I thought as a workaround I could import the file in my journey but when I do this I get the size exceeded error.

4

@joel.parker can you please share the script when you were trying to use and getting an error?

5

Sure, it's worth noting too that I am using a slight variation of pdf-parse called pdf-parse-debugging-disabled due to a different error I had earlier.

This is my journey, I have looped it for checking different values but I do get the same push error when I try to just push this journey without the loop.

const { journey } = require("@elastic/synthetics");
import { environmentUrls } from "../../utils/scripts/environmentUrls";
import { loadPageCookies } from "../../utils/pageObjectModel/common/cookies";
import { getBrandPrefix } from "../../utils/scripts/getBrandPrefix";
import { assertWarrantyLightBoxIsCorrect } from "../../utils/pageObjectModel/productPage";
import { assertPdfContentIsCorrect } from "../../utils/pageObjectModel/productPage";

const brands = ["very"];
const insuranceProducts = ["repair","replace","screen"];
const documentTypes = ["ipid","termsConditions","furtherRelevantInfo"];
const products =
{
  "repair": "/1600406124.prd",
  "repair_monthly": "/1600406124.prd",
  "replace": "/1600488041.prd",
  "furniture": "/1600292792.prd",
  "screen": "/1600930565.prd"
};

for (const brand of brands) {
  for (const documentType of documentTypes) {
    for (const insuranceProduct of insuranceProducts) {
      const product = products[insuranceProduct];
      if ((insuranceProduct == 'furniture' || insuranceProduct == 'screen') && (documentType == 'furtherRelevantInfo')) {
        break;
      };
      journey(

        {
          name: `${getBrandPrefix(brand)} | FS | ` + insuranceProduct + ` ` + documentType + ` PDF contains the correct text `,
          tags: ["fs", brand],
        },
        async ({ page }) => {
          loadPageCookies(page, environmentUrls[brand].baseUrl + product);
          assertPdfContentIsCorrect(page, brand, documentType, insuranceProduct);
        }
      )

    }
  }
}

This is the productPage.js file where I am importing and using the pdf-parse module.

`import pdf from "pdf-parse-debugging-disabled"
import pdf2 from "pdf-parse-debugging-disabled/lib/pdf.js/v1.10.100/build/pdf"


import { furnitureIpidText, repairIpidText, repair_monthlyIpidText, replaceIpidText, screenIpidText } from "../../resources/very/pdfText/ipidText";
import { downloadInsurancePdfDocument } from "../handlers/downloadHandler";
import { repairLightboxText, repair_monthlyLightboxText, replaceLightboxText, screenLightboxText, furnitureLightboxText } from "../../resources/very/insuranceText/lightboxText/lightboxText";
import { furnitureTermsConditionsText, repairTermsConditionsText, repair_monthlyTermsConditionsText, replaceTermsConditionsText, screenTermsConditionsText } from "../../resources/very/pdfText/termsConditionsText";
import { repairFurtherRelevantInfoText, repair_monthlyFurtherRelevantInfoText, replaceFurtherRelevantInfoText } from "../../resources/very/pdfText/furtherRelevantInfoText";

const { step, expect } = require("@elastic/synthetics");

const selectors = {
  productOptions: {
    productOptionsList: ".productOptionsList",
    warrantyFindOutMore: ".warrantyFindOutMore"
  },
  warrantyLightbox: {
    warrantyOptionExplained: ".warrantyOptionExplained",
    warrantyLightBoxText: ".productWarrantyContent",
    monthlyRepairLightboxText: ".monthlySection",
    standardRepairLightBoxText: ".standardSection",
    ipidDownloadLink: "",
    fixedTermInsuranceTab: "#fixedTab"
  }

};

module.exports.assertWarrantyLightBoxIsCorrect = function (page, insuranceProduct) {
  step("Click on find out more in the warranty section", async () => {
    expect((await page.$(selectors.productOptions.warrantyFindOutMore)) !== null);
    await page.locator(selectors.productOptions.warrantyFindOutMore).click();
    await page.locator(selectors.warrantyLightbox.warrantyOptionExplained).isVisible();
  });
  step("Assert that the actual lightbox text matches the expected text", async () => {
    var actualText;
    var expectedText;
    switch (insuranceProduct) {
      case 'repair': expectedText = repairLightboxText
        break;
      case 'repair_monthly': expectedText = repair_monthlyLightboxText
        break;
      case 'replace': expectedText = replaceLightboxText
        break;
      case 'screen': expectedText = screenLightboxText
        break;
      case 'furniture': expectedText = furnitureLightboxText;
        break;
    }
    if (insuranceProduct == 'repair') {
      actualText = await page.locator(selectors.warrantyLightbox.standardRepairLightBoxText).first().textContent();
    } else if (insuranceProduct == 'repair_monthly') {
      actualText = await page.locator(selectors.warrantyLightbox.monthlyRepairLightboxText).first().textContent();
    } else {
      actualText = await page.locator(selectors.warrantyLightbox.warrantyLightBoxText).first().textContent();
    }

    expect(actualText.replace(/\s+/g, '')).toEqual(expectedText.replace(/\s+/g, ''));
  })
}

module.exports.assertPdfContentIsCorrect = function (page, brand, documentType, insuranceProduct) {
  step("Click on find out more in the warranty section", async () => {
    expect((await page.$(selectors.productOptions.warrantyFindOutMore)) !== null);
    await page.locator(selectors.productOptions.warrantyFindOutMore).click();
    await page.locator(selectors.warrantyLightbox.warrantyOptionExplained).isVisible();
    // if (insuranceProduct == 'repair') {
    //   await page.locator(selectors.warrantyLightbox.fixedTermInsuranceTab).click();
    // }
  });
  step("Download the PDF and check it matches the expected document for: " + documentType, async () => {
    const expectedPdfPath = "resources/" + brand + "/" + documentType + "/" + insuranceProduct + "_" + documentType + ".pdf";
    var expectedPdf;

    switch (insuranceProduct) {
      case 'repair':
        if (documentType == 'ipid') expectedPdf = repairIpidText;
        if (documentType == 'termsConditions') expectedPdf = repairTermsConditionsText;
        if (documentType == 'furtherRelevantInfo') expectedPdf = repairFurtherRelevantInfoText;
        break;
      case 'repair_monthly':
        if (documentType == 'ipid') expectedPdf = repair_monthlyIpidText;
        if (documentType == 'termsConditions') expectedPdf = repair_monthlyTermsConditionsText;
        if (documentType == 'furtherRelevantInfo') expectedPdf = repair_monthlyFurtherRelevantInfoText;
        break;
      case 'replace':
        if (documentType == 'ipid') expectedPdf = replaceIpidText;
        if (documentType == 'termsConditions') expectedPdf = replaceTermsConditionsText;
        if (documentType == 'furtherRelevantInfo') expectedPdf = replaceFurtherRelevantInfoText;
        break;
      case 'screen':
        if (documentType == 'ipid') expectedPdf = screenIpidText;
        if (documentType == 'termsConditions') expectedPdf = screenTermsConditionsText;
        break;
      case 'furniture':
        if (documentType == 'ipid') expectedPdf = furnitureIpidText;
        if (documentType == 'termsConditions') expectedPdf = furnitureTermsConditionsText;
        break;
    }
    var result = await downloadAndComparePdf(page,formatDocumentType(documentType),expectedPdf);
    expect(result).toEqual(true);
  })
}

const formatDocumentType = function (documentType) {
  switch (documentType) {
    case 'ipid': return 'IPID';
    case 'termsConditions': return 'Terms and Conditions';
    case 'furtherRelevantInfo': return 'Further Relevant Information';
  }
};

const downloadAndComparePdf = async function (page, documentType, expectedPdf) {
  try {
    const downloadPromise = page.waitForEvent('download', { timeout: 5000 }); // Option 1: Increased timeout
    await page.getByRole("link")
      .filter({ hasText: documentType })
      .click();
    await downloadPromise; // Wait for the download event to complete
    const download = await downloadPromise; // Reusing the promise, as it already resolved
    const readStream = await download.createReadStream();

    readStream.on('end', () => {
      console.log("Stream ended");
    });

    readStream.on('error', (error) => {
      console.error('Error in stream:', error);
      return 'error in stream';
    });

    // Convert the stream to a buffer
    const chunks = [];
    for await (const chunk of readStream) {
      chunks.push(chunk);
    }
    const pdfBuffer = Buffer.concat(chunks);

    // Parse the PDF buffer
    
    const data = await pdf(pdfBuffer);
    if(data.text.replace(/\s+/g, '') == expectedPdf.replace(/\s+/g, '')){
      return true;
    }    
    else{
      return 'PDF didnt match';
    }
  } catch (error) {
    console.error("Error during PDF parsing:", error.message || error);
    return error.message;
  }

};

`
6

@joel.parker wanted to let you know that we were able to reproduce the issue. The team will work on it and will get back to you if there's an update.

7

Hi @joel.parker,

Thanks for sharing the code, As a interim solution we will double the limits on the Synthetics side - feat: double monitor bundle size to 1500Kb by vigneshshanmugam · Pull Request #882 · elastic/synthetics · GitHub and we will work on a better approach to enable minification for the final bundles to reduce the footprint even lower - Add minification support for project based monitors before pushed to Kibana · Issue #883 · elastic/synthetics · GitHub

Will update here once we release the new agent.

Thanks,
Vignesh

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.