Continuous transform not working as expected

Elastic Stack Elasticsearch
1

Hello,

I have problem that my continuous transform doesn't update as the index, which it takes data from, is getting new and new data.

The original index, let's name it playwright-test-runs-data (as i have it named), is getting json which consists of array of objects looking like this:

{
          "testName" : "Some test name",
          "numberOfRuns" : 1,
          "numberOfFails" : 0,
          "testStatus" : "expected",
          "environment" : "master",
          "browser" : "chromium",
          "testType" : "fullSuite",
          "dateTimestamp" : 1670927212470
}

On this playwright-test-runs-data i have created a Kibana index with a same name, where I added a field named timestamp with date type (cause i wasn't able to successfully send a string that would match automatic date detection) that takes the dateTimestamp field and converts it into a date string format.

On this Kibana playwright-test-runs-data i run continuous transform with the timestamp field, as the field it checks for new data, which has this grouping and aggregations:

{
  "group_by": {
    "timestamp": {
      "date_histogram": {
        "field": "timestamp",
        "calendar_interval": "1m"
      }
    },
    "testName": {
      "terms": {
        "field": "testName.keyword"
      }
    },
    "browser": {
      "terms": {
        "field": "browser.keyword"
      }
    },
    "environment": {
      "terms": {
        "field": "environment.keyword"
      }
    },
    "testStatus": {
      "terms": {
        "field": "testStatus.keyword"
      }
    },
    "testType": {
      "terms": {
        "field": "testType.keyword"
      }
    }
  },
  "aggregations": {
    "numberOfFails.sum": {
      "sum": {
        "field": "numberOfFails"
      }
    },
    "numberOfRuns.sum": {
      "sum": {
        "field": "numberOfRuns"
      }
    }
  }
}

After i create and start the transform named "playwright-test-runs-transformed-data", it takes all data from the original index and transforms it all, but after that it doesn't update at all if the "playwright-test-runs-data" index gets new data, as can be seen in the screenshots

image
image2004×166 38.7 KB

image
image1970×86 20.8 KB

Can you please help me find some mistake i made along the way or if I am missing some settings needed to be set?

If you will need more information, I will have no problem providing it if it will be possible.

2

Hi,

Could you please share the full transform config?
You can get it from the UI by expanding the row in the transforms table and selecting the JSON tab.

Thanks,
James

3

Okay, that's what i found

value={
  "id": "playwright-test-runs-transformed-data",
  "version": "7.17.1",
  "create_time": 1670883112635,
  "source": {
    "index": [
      "playwright-test-runs-data*"
    ],
    "query": {
      "match_all": {}
    },
    "runtime_mappings": {
      "timestamp": {
        "type": "date",
        "script": {
          "source": "def timestamp = doc['dateTimestamp'].value;\nemit(timestamp);"
        }
      }
    }
  },
  "dest": {
    "index": "playwright-test-runs-transformed-data"
  },
  "frequency": "1m",
  "sync": {
    "time": {
      "field": "timestamp",
      "delay": "50s"
    }
  },
  "pivot": {
    "group_by": {
      "timestamp": {
        "date_histogram": {
          "field": "timestamp",
          "calendar_interval": "1m"
        }
      },
      "testName": {
        "terms": {
          "field": "testName.keyword"
        }
      },
      "environment": {
        "terms": {
          "field": "environment.keyword"
        }
      },
      "browser": {
        "terms": {
          "field": "browser.keyword"
        }
      },
      "testStatus": {
        "terms": {
          "field": "testStatus.keyword"
        }
      },
      "testType": {
        "terms": {
          "field": "testType.keyword"
        }
      }
    },
    "aggregations": {
      "numberOfRuns.sum": {
        "sum": {
          "field": "numberOfRuns"
        }
      },
      "numberOfFails.sum": {
        "sum": {
          "field": "numberOfFails"
        }
      }
    }
  },
  "settings": {
    "max_page_search_size": 500
  }
}

Is that correct?

4

Hello @James_Gowdy,

I hope i sent the right data you wanted. If you have a time, could you please have a look on the config?

Thanks,
Dominik

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.