# Rollup Job - New data not Rolled Up

**URL:** <https://discuss.elastic.co/t/rollup-job-new-data-not-rolled-up/330006>\
**Category:** Kibana\
**Tags:** rollups\
**Created:** [April 14, 2023, 11:37am UTC](https://discuss.elastic.co/t/rollup-job-new-data-not-rolled-up/330006 "2023-04-14T11:37:20Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Vitor\_Nilson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vitor_nilson/32/119824_2.png) [@Vitor\_Nilson](https://discuss.elastic.co/u/Vitor_Nilson)\
**Post date:** [April 14, 2023, 11:37am UTC](https://discuss.elastic.co/t/rollup-job-new-data-not-rolled-up/330006/1 "2023-04-14T11:37:21Z")

</div>

Hello,

I'm currently creating a rollup job to reduce storage costs in my company.

The rollup job worked well, grouped by the fields that I specified. The problem is that only data older than 7 days is being rolled up to my job even if the Delay is None or 1 minute.

This is my job configuration:

```auto
{
  "jobs": [
    {
      "config": {
        "id": "gateway_kong_rollup_job",
        "index_pattern": "kong",
        "rollup_index": "rollup_job_kong_gateway",
        "cron": "0 * * * * ?",
        "groups": {
          "date_histogram": {
            "fixed_interval": "24h",
            "field": "@timestamp",
            "time_zone": "UTC"
          },
          "terms": {
            "fields": [
              "field_1",
              "field_2",
              "field_3",
              "field_4"
            ]
          }
        },
        "metrics": [],
        "timeout": "20s",
        "page_size": 1000
      },
      "status": {
        "job_state": "started",
        "current_position": {
          "@timestamp.date_histogram": 1680739200000,
          "field_1": "test",
          "field_2": "test",
          "field_3": "test",
          "field_4": "test"
        }
      },
      "stats": {
        "pages_processed": 1285,
        "documents_processed": 99779,
        "rollups_indexed": 305,
        "trigger_count": 1284,
        "index_time_in_ms": 250,
        "index_total": 1,
        "index_failures": 0,
        "search_time_in_ms": 1893,
        "search_total": 1285,
        "search_failures": 0,
        "processing_time_in_ms": 4,
        "processing_total": 1285
      }
    }
  ]
}

```

As you can see, the `current_position` of the Index is at timestamp `1680739200000` Which, converted, refers to `Thursday, 6 April 2023 00:00:00`.

The `"index_pattern": "kong"`, which my my rollup job is looking for, has data for today (2023-04-14):

**Index Kong Discover View:**

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/d/4d929797ed19a0a27e649a6819993e85208a07bc.png)

(There's a lack of data between 2023-04-07 and 2023-04-13, don't worry about this.)

As my job is running once per minute, it's supposed to have the data for today. But this is not happening.

**My rollup job Discover View:**

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/f/5ffadc1d8ef311b1dc4f3e9e4f35bcc0ff39978d.png)

As you can see, there's no data for 2023-04-13 and 2023-04-14.

There's a way that I can force the job to get this new data or change its `current_position`?

---

<div class="post-metadata">

**Author:** ![dawn](https://avatars.discourse-cdn.com/v4/letter/d/71e660/32.png) [@dawn](https://discuss.elastic.co/u/dawn)\
**Post date:** [April 14, 2023, 5:53pm UTC](https://discuss.elastic.co/t/rollup-job-new-data-not-rolled-up/330006/2 "2023-04-14T17:53:25Z")

</div>

I have a similar issue with a Transform. Interested to see if there is a solution to your problem, which may also help me with my issue.

---

<div class="post-metadata">

**Author:** ![Marius\_Dragomir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_dragomir/32/42087_2.png) [@Marius\_Dragomir](https://discuss.elastic.co/u/Marius_Dragomir)\
**Post date:** [April 15, 2023, 1:24am UTC](https://discuss.elastic.co/t/rollup-job-new-data-not-rolled-up/330006/3 "2023-04-15T01:24:38Z")

</div>

I will have to refer you to the Elasticsearch part of the forum, because all that Kibana does is give the UI that sends the request for a rollup job to start to Elasticsearch. That requests looks ok from what I can see, the cron interval should not a problem. Does it show something else in the Kibana Rollups UI?

---

<div class="post-metadata">

**Author:** ![Vitor\_Nilson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vitor_nilson/32/119824_2.png) [@Vitor\_Nilson](https://discuss.elastic.co/u/Vitor_Nilson)\
**Post date:** [April 19, 2023, 11:17am UTC](https://discuss.elastic.co/t/rollup-job-new-data-not-rolled-up/330006/4 "2023-04-19T11:17:27Z")

</div>

Guys, thanks for the responses.

It turns out that someone at one point changed the name of the logstash index and did not inform anyone.

The only way to figure this out was to look at the index which should have been working fine and see that no data was coming in.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 17, 2023, 11:18am UTC](https://discuss.elastic.co/t/rollup-job-new-data-not-rolled-up/330006/5 "2023-05-17T11:18:15Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.