# How to configure ANOMALY DETECTION with DAILY buckets **URL:** \ **Category:** Kibana\ **Tags:** elastic-stack-machine-learning\ **Created:** [January 31, 2020, 6:47pm UTC](https://discuss.elastic.co/t/how-to-configure-anomaly-detection-with-daily-buckets/217451 "2020-01-31T18:47:42Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![alexisdrnt](https://avatars.discourse-cdn.com/v4/letter/a/898d66/32.png) [@alexisdrnt](https://discuss.elastic.co/u/alexisdrnt)\ **Post date:** [January 31, 2020, 6:47pm UTC](https://discuss.elastic.co/t/how-to-configure-anomaly-detection-with-daily-buckets/217451/1 "2020-01-31T18:47:43Z")
Hi, I am having a hard trying to figure out how to setup the datafeed of my Anomaly Detection Job. I have a Transform job that aggregate all the daily data to the same day at 00:00. I wanna run a Anomaly Detection to run onto this index. It is a daily bucket (`bucket_span: 1d`) If I understand well the query delay should be 24h (`query_delay: 24h`), since I wanna wait until the end of the day for all the data of the same day to be aggregated at 00:00 this day. And the frequency is set to 24h as well, since I only have one timestamp a day (`frequency: 24h`). However it doesn't work. Let's say I got data for 2020-01-29 all aggregated at 00:00, I will be expecting the ML job the process this document at 2020-01-30 00:00 but it does not work. I have to wait until 2020-01-30 00:00 for the data of January 29th to be processed. Any idea? P.S.: All the date/time are expressed in UTC ---
**Author:** ![Hendrik\_Muhs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendrik_muhs/32/25802_2.png) [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)\ **Post date:** [February 3, 2020, 10:04am UTC](https://discuss.elastic.co/t/how-to-configure-anomaly-detection-with-daily-buckets/217451/2 "2020-02-03T10:04:35Z")
Hi, interesting combination of transforms and anomaly detection. I think for configuring the `query_delay` you also have to take additional delays of the transform into account. As I understand your transform runs every 24 hours, but results will not be immediately available. The transform needs time to process all data and persist the results. It seems that transform and anomaly detection run at the same time, but anomaly detection should run _after_ transform. The query delay should therefore add a little extra. Do you know how long it takes to transform the data of the last 24 hours? You should be able to find out by checking the audit information (job messages) in the UI, it sends a audit message after the checkpoint has finished. If it took e.g. 5 minutes you should add these 5 minutes to the 24 hours plus a little extra. Regarding the anomaly detector configuration: Your configuration for `frequency` would still only trigger anomaly detection every 24 hours and it would run it probably too early. I suggest to lower it. e.g. to `1h`, so your should get results 1 at 01:00. The unnecessary triggers till the next full run shouldn't be a big problem. We will discuss this issue in the engineering team, to see how we can improve. ---
**Author:** ![Hendrik\_Muhs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendrik_muhs/32/25802_2.png) [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)\ **Post date:** [February 3, 2020, 11:58am UTC](https://discuss.elastic.co/t/how-to-configure-anomaly-detection-with-daily-buckets/217451/3 "2020-02-03T11:58:50Z")
FWIW, could you describe your usecase in more detail? What does the transform do, what anomalies are you looking for? Data feeds can run aggregations on their own, I wonder whether the transform is really needed. Additionally a bucket span of 24 hours is unusual for an anomaly detection usecase. ---
**Author:** ![alexisdrnt](https://avatars.discourse-cdn.com/v4/letter/a/898d66/32.png) [@alexisdrnt](https://discuss.elastic.co/u/alexisdrnt)\ **Post date:** [February 3, 2020, 3:55pm UTC](https://discuss.elastic.co/t/how-to-configure-anomaly-detection-with-daily-buckets/217451/4 "2020-02-03T15:55:59Z")
Hey @Hendrik_Muhs, - I set up a transform job with a `date histogram` set to **1d**. All the data coming during a given day is gonna be aggregated to the same day at midnight. e.g.: Let's say I have data coming in at timestamp **2020-02-03 08:00:00** and another record coming in at timestamp **2020-02-03 23:59:59** , both of them are gonna be aggregated at timestamp **2020-02-03 00:00:00** (The Transform `frequency` is **60s** ). - Therefor I wanna wait 24h before the Anomaly Job picks up the data. e.g.: The job running at timestamp **2020-02-04 00:00:00** should pick up the records at timestamp **2020-02-03 00:00:00**. Setting the `frequency` to `24h` and the `query_delay` to `24h`: the Anomaly Job at timestamp **2020-02-04 00:00:00** is not gonna pick up the records at **2020-02-03 00:00:00** , for that I'll have to wait until timestamp **2020-02-05 00:00:00** for it to be picked up. Regarding the aggregation in the Transform job, I need to `divide` the `SUM` of two field. Is that something possible in the datafeed directly? let's say I have two records like this: `{_id: 1, field_1: 3, field_2: 5} and {_id: 2, field_1: 7, field_2: 1}` I need to create a field like so:` (3+7) / (5+1)` ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [February 3, 2020, 5:19pm UTC](https://discuss.elastic.co/t/how-to-configure-anomaly-detection-with-daily-buckets/217451/5 "2020-02-03T17:19:57Z")
Yes, you can accomplish this via a `script_field`. See this example: [ML Job on Scripted field](https://discuss.elastic.co/t/ml-job-on-scripted-field/116820/14) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 2, 2020, 5:19pm UTC](https://discuss.elastic.co/t/how-to-configure-anomaly-detection-with-daily-buckets/217451/6 "2020-03-02T17:19:58Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.