Set custom Transform schedule time in kibana

[dash5000 (Dash)]

Hello Everyone,

I just started using transform in kibana UI to aggregate data and also to have a longer retention period for visualization. The data collections work fine, my transform basically collectors two fields which are aggregated on max.

My transform json is

{
  "id": "testu",
  "version": "7.16.1",
  "create_time": 1646254133114,
  "source": {
    "index": [
      "test12-*"
    ],
    "query": {
      "match_all": {}
    }
  },
  "dest": {
    "index": "test123"
  },
  "frequency": "1h",
  "sync": {
    "time": {
      "field": "@timestamp",
      "delay": "60s"
    }
  },
  "pivot": {
    "group_by": {
      "@timestamp": {
        "date_histogram": {
          "field": "@timestamp",
          "calendar_interval": "1d"
        }
      },
      "device": {
        "terms": {
          "field": "devndevice"
        }
      },
      "name": {
        "terms": {
          "field": "devname"
        }
      }
    },
    "aggregations": {
      "in-use.max": {
        "max": {
          "field": "in-use"
        }
      },
      "out-use.max": {
        "max": {
          "field": "out-use"
        }
      }
    }
  },
  "settings": {
    "max_page_search_size": 500
  },
  "retention_policy": {
    "time": {
      "field": "@timestamp",
      "max_age": "180d"
    }
  }
}

The issue is that the my data collections happens at 7 in the evening but i want that to happen at midnight. Also since i am having a timestamp for 24h because i want to know the max for both fields each day till the previous day.

For example: March 3 12:05 AM should show run the transform and update till the march March 2 day .

How could i configure the transform to perform on a schedule time everyday to get the daily peaks of the two fields till the previous day. Currently my transform run every day at evening 7 and it generates till the previous day till the same time.
Just want to schedule the time to early morning at 12:10 from 7PM.

Regards,
Darshan

[Hendrik_Muhs]

Your transform runs every hour, because frequency is set to 1h. What you are experiencing is not a problem of scheduling, but a timezone issue. Elasticsearch uses the UTC time, to have a consistent time allover the world. All dates are stored in UTC and the date_histogram will produce buckets according to UTC. Transform now runs shortly after midnight, but it is midnight according to UTC, not midnight according to your timezone.

You can change the timezone of your date_histogram. This will tell the aggregation to create buckets according to the specified timezone.

Transform will still check every hour, but now start shortly after midnight according to the configured timezone.

FWIW: The behavior is by design, if a date_histogram is configured transform executes - at default - when it retrieved all data for a bucket. You can change the default in settings by setting align_checkpoints to false. This will produce intermediate results, so you can see the data of the current day while the day isn't over yet. However that is obviously more expensive as it re-calculates the buckets more often. That's why align_checkpoints has the default setting true.

[dash5000 (Dash)]

Thanks @Hendrik_Muhs for the answer. One more thing, i wanted to ask is when i am visualizing the data of the transform in a line lens. I have two fields in group_by block devndevice and devname, I need to display two lines for same aggregation but from different devname in same visualization using kibana. Currently its calculating the max of all the the same field across all devname and visualizing a single line. Is there any way that i can get multiple line with the same field with different devname.

FOr example: we have devname=test1 devndevice=test, and devname=test2 devndevice=test, I should be able to visualize both the metrics for different devndevice not just the single one that is combining both devndeivce.

Regards,
Darshan

[Hendrik_Muhs]

I am sorry, I can't answer the last question. From a data perspective it should work, because you group by devname in addition to date_histogram.

I hope someone else can answer. If you should not get an answer after some waiting, please open a new thread.