Deleted Rollup Job but cannot re-create it

Elastic Stack Elasticsearch
1

After creating a rollup job and deleting it we are unable to create a job again with the same id.

POST /_xpack/rollup/job/foo
{
    "error": {
        "root_cause": [
            {
                "type": "status_exception",
                "reason": "Cannot create rollup job [foo] because job was previously created (existing metadata)."
            }
        ],
        "type": "status_exception",
        "reason": "Cannot create rollup job [foo] because job was previously created (existing metadata)."
    },
    "status": 409
}

We've read through the docs but the following does not help:

POST /rollup-test/_delete_by_query
{
"query": {
    "term": {
    "_rollup.id": "foo"
    }
}
}

Here is the creation script:

POST /_xpack/rollup/job/foo
{
    "index_pattern": "access-test",
    "rollup_index": "rollup-test",
    "cron": "0 0 1 * * ?",
    "page_size": 100,
    "groups":  {
    "date_histogram": {
        "field": "timestamp",
        "interval": "1d"
    },
    "terms": {
        "fields": ["category", "subcategory", "method"]
    }
    },
    "metrics": [
        {
            "field": "timeTaken",
            "metrics": ["min", "max", "avg"]
        }
    ]
}

We want to delete and re-create the job because there doesn't seem to be a way to change/update a rollup job once it's created. Only GET, CREATE and DELETE.

How can we delete the metadata so we can re-create the job?

2

The rollup data is not removed when a job is deleted so you need to delete the data manually or use a different rollup_index.

3

Is there any other data in the rollup index, or does it hold only a single rollup job? If it's a single job, and you don't want the job anymore, it's simplest to just delete the entire rollup index as @jimczi mentioned

If the index contains multiple jobs, and you want to reuse it, you'll need to edit the _meta field of the index using a Put Mapping API. Basically, you'll need to remove the job's metadata from the _meta, which is what we use to determine what jobs a particular index holds.

This is pretty user-unfriendly and prone to error. :confused: We have a ticket open to enhance the DeleteJob API to do this for you: [Rollup] Add ability to delete rollup data · Issue #31347 · elastic/elasticsearch · GitHub

4

But where is the data? We did a POST /rollup-test/_delete_by_query so that index was empty (docs deleted). We also cannot re-create foo even if foo never ran and had no documents. Currently we have to re-create foo1, foo2, foo3 each time we need to make a change.

5

@cawoodm see my explanation above. The _meta on the index is the issue here.

For example, if I create a job:

PUT _xpack/rollup/job/foo
{
  "index_pattern": "foo",
  "rollup_index": "rollup",
  "cron": "*/30 * * * * ?",
  "page_size": 10,
  "groups": {
    "date_histogram": {
      "field": "timestamp",
      "interval": "1h"
    },
    "terms": {
      "fields": [
        "partition"
      ]
    }
  },
  "metrics": [
    {
      "field": "price",
      "metrics": [
        "max"
      ]
    }
  ]
}

And go look at the index that is created, you'll see that it contains metadata on the index:

GET rollup/

{
  "rollup": {
    "aliases": {},
    "mappings": {
      "_doc": {
        "_meta": {
          "_rollup": {
            "foo": {
              "cron": "*/30 * * * * ?",
              "rollup_index": "rollup",
              "groups": {
                "date_histogram": {
                  "field": "timestamp",
                  "interval": "1h",
                  "time_zone": "UTC"
                },
                "terms": {
                  "fields": [
                    "partition"
                  ]
                }
              },
              "id": "foo",
              "metrics": [
                {
                  "field": "price",
                  "metrics": [
                    "max"
                  ]
                }
              ],
              "index_pattern": "foo",
              "timeout": "20s",
              "page_size": 10
            }
          },
          "rollup-version": "7.0.0-alpha1"
        },
        "dynamic_templates": [
          {
            "strings": {
              "match_mapping_type": "string",
              "mapping": {
                "type": "keyword"
              }
            }
          },
          {
            "date_histograms": {
              "path_match": "*.date_histogram.timestamp",
              "mapping": {
                "type": "date"
              }
            }
          }
        ]
      }
    },
    "settings": {
      "index": {
        "creation_date": "1535565697662",
        "number_of_shards": "1",
        "number_of_replicas": "1",
        "uuid": "5NaceDtdSKiw92VR6gEqqQ",
        "version": {
          "created": "7000001"
        },
        "provided_name": "rollup"
      }
    }
  }
}

The metadata on the index is what Rollup checks to make sure we don't duplicate jobs/recreate a job with different settings. The delete-by-query doesn't remove the metadata, which is the cause of the issues you're seeing.

If you don't care about the data in the rollup index (e.g. it's test data, or empty) just delete the index itself after deleting the job:

DELETE _xpack/rollup/job/foo
DELETE rollup/

And then you can recreate the job:

PUT _xpack/rollup/job/foo
{
  "index_pattern": "foo",
  "rollup_index": "rollup",
  "cron": "*/30 * * * * ?",
  "page_size": 10,
  "groups": {
    "date_histogram": {
      "field": "timestamp",
      "interval": "1h"
    },
    "terms": {
      "fields": [
        "partition"
      ]
    }
  },
  "metrics": [
    {
      "field": "price",
      "metrics": [
        "max"
      ]
    }
  ]
}

{
  "acknowledged": true
}
1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.