Continuous transform not working as expected

Hello,

I have problem that my continuous transform doesn't update as the index, which it takes data from, is getting new and new data.

The original index, let's name it playwright-test-runs-data (as i have it named), is getting json which consists of array of objects looking like this:

{
          "testName" : "Some test name",
          "numberOfRuns" : 1,
          "numberOfFails" : 0,
          "testStatus" : "expected",
          "environment" : "master",
          "browser" : "chromium",
          "testType" : "fullSuite",
          "dateTimestamp" : 1670927212470
}

On this playwright-test-runs-data i have created a Kibana index with a same name, where I added a field named timestamp with date type (cause i wasn't able to successfully send a string that would match automatic date detection) that takes the dateTimestamp field and converts it into a date string format.

On this Kibana playwright-test-runs-data i run continuous transform with the timestamp field, as the field it checks for new data, which has this grouping and aggregations:

{
  "group_by": {
    "timestamp": {
      "date_histogram": {
        "field": "timestamp",
        "calendar_interval": "1m"
      }
    },
    "testName": {
      "terms": {
        "field": "testName.keyword"
      }
    },
    "browser": {
      "terms": {
        "field": "browser.keyword"
      }
    },
    "environment": {
      "terms": {
        "field": "environment.keyword"
      }
    },
    "testStatus": {
      "terms": {
        "field": "testStatus.keyword"
      }
    },
    "testType": {
      "terms": {
        "field": "testType.keyword"
      }
    }
  },
  "aggregations": {
    "numberOfFails.sum": {
      "sum": {
        "field": "numberOfFails"
      }
    },
    "numberOfRuns.sum": {
      "sum": {
        "field": "numberOfRuns"
      }
    }
  }
}

After i create and start the transform named "playwright-test-runs-transformed-data", it takes all data from the original index and transforms it all, but after that it doesn't update at all if the "playwright-test-runs-data" index gets new data, as can be seen in the screenshots


Can you please help me find some mistake i made along the way or if I am missing some settings needed to be set?

If you will need more information, I will have no problem providing it if it will be possible.

Hi,

Could you please share the full transform config?
You can get it from the UI by expanding the row in the transforms table and selecting the JSON tab.

Thanks,
James

Okay, that's what i found

value={
  "id": "playwright-test-runs-transformed-data",
  "version": "7.17.1",
  "create_time": 1670883112635,
  "source": {
    "index": [
      "playwright-test-runs-data*"
    ],
    "query": {
      "match_all": {}
    },
    "runtime_mappings": {
      "timestamp": {
        "type": "date",
        "script": {
          "source": "def timestamp = doc['dateTimestamp'].value;\nemit(timestamp);"
        }
      }
    }
  },
  "dest": {
    "index": "playwright-test-runs-transformed-data"
  },
  "frequency": "1m",
  "sync": {
    "time": {
      "field": "timestamp",
      "delay": "50s"
    }
  },
  "pivot": {
    "group_by": {
      "timestamp": {
        "date_histogram": {
          "field": "timestamp",
          "calendar_interval": "1m"
        }
      },
      "testName": {
        "terms": {
          "field": "testName.keyword"
        }
      },
      "environment": {
        "terms": {
          "field": "environment.keyword"
        }
      },
      "browser": {
        "terms": {
          "field": "browser.keyword"
        }
      },
      "testStatus": {
        "terms": {
          "field": "testStatus.keyword"
        }
      },
      "testType": {
        "terms": {
          "field": "testType.keyword"
        }
      }
    },
    "aggregations": {
      "numberOfRuns.sum": {
        "sum": {
          "field": "numberOfRuns"
        }
      },
      "numberOfFails.sum": {
        "sum": {
          "field": "numberOfFails"
        }
      }
    }
  },
  "settings": {
    "max_page_search_size": 500
  }
}

Is that correct?

Hello @James_Gowdy,

I hope i sent the right data you wanted. If you have a time, could you please have a look on the config?

Thanks,
Dominik

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.