Rollup Job - New data not Rolled Up

[Vitor_Nilson]

Hello,

I'm currently creating a rollup job to reduce storage costs in my company.

The rollup job worked well, grouped by the fields that I specified. The problem is that only data older than 7 days is being rolled up to my job even if the Delay is None or 1 minute.

This is my job configuration:

{
  "jobs": [
    {
      "config": {
        "id": "gateway_kong_rollup_job",
        "index_pattern": "kong",
        "rollup_index": "rollup_job_kong_gateway",
        "cron": "0 * * * * ?",
        "groups": {
          "date_histogram": {
            "fixed_interval": "24h",
            "field": "@timestamp",
            "time_zone": "UTC"
          },
          "terms": {
            "fields": [
              "field_1",
              "field_2",
              "field_3",
              "field_4"
            ]
          }
        },
        "metrics": [],
        "timeout": "20s",
        "page_size": 1000
      },
      "status": {
        "job_state": "started",
        "current_position": {
          "@timestamp.date_histogram": 1680739200000,
          "field_1": "test",
          "field_2": "test",
          "field_3": "test",
          "field_4": "test"
        }
      },
      "stats": {
        "pages_processed": 1285,
        "documents_processed": 99779,
        "rollups_indexed": 305,
        "trigger_count": 1284,
        "index_time_in_ms": 250,
        "index_total": 1,
        "index_failures": 0,
        "search_time_in_ms": 1893,
        "search_total": 1285,
        "search_failures": 0,
        "processing_time_in_ms": 4,
        "processing_total": 1285
      }
    }
  ]
}

As you can see, the current_position of the Index is at timestamp 1680739200000 Which, converted, refers to Thursday, 6 April 2023 00:00:00.

The "index_pattern": "kong", which my my rollup job is looking for, has data for today (2023-04-14):

Index Kong Discover View:

(There's a lack of data between 2023-04-07 and 2023-04-13, don't worry about this.)

As my job is running once per minute, it's supposed to have the data for today. But this is not happening.

My rollup job Discover View:

As you can see, there's no data for 2023-04-13 and 2023-04-14.

There's a way that I can force the job to get this new data or change its current_position?

[dawn]

I have a similar issue with a Transform. Interested to see if there is a solution to your problem, which may also help me with my issue.

[Marius_Dragomir]

I will have to refer you to the Elasticsearch part of the forum, because all that Kibana does is give the UI that sends the request for a rollup job to start to Elasticsearch. That requests looks ok from what I can see, the cron interval should not a problem. Does it show something else in the Kibana Rollups UI?

[Vitor_Nilson]

Guys, thanks for the responses.

It turns out that someone at one point changed the name of the logstash index and did not inform anyone.

The only way to figure this out was to look at the index which should have been working fine and see that no data was coming in.