# Transforms keep falling behind / not processing any documents

**URL:** <https://discuss.elastic.co/t/transforms-keep-falling-behind-not-processing-any-documents/363596>\
**Category:** Elasticsearch\
**Tags:** docker\
**Created:** [July 23, 2024, 1:23am UTC](https://discuss.elastic.co/t/transforms-keep-falling-behind-not-processing-any-documents/363596 "2024-07-23T01:23:28Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Liam\_Young](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/liam_young/32/131373_2.png) [@Liam\_Young](https://discuss.elastic.co/u/Liam_Young)\
**Post date:** [July 23, 2024, 1:23am UTC](https://discuss.elastic.co/t/transforms-keep-falling-behind-not-processing-any-documents/363596/1 "2024-07-23T01:23:28Z")

</div>

Hi, I have a datastream that receives 1000s of requests a minute. I am testing with a script to make logs, it makes 1000 requests in a minute. I am very new to Elasticsearch. The datastream works fine. I set up a transform (1\_minute) to group and aggregate these logs by 1 minute along with other groupings, this works perfectly. I then set up another transform with the source pointed to the 1\_minute destination index. Here I group it into 1\_hour instead of 1 minute, with the same mappings overall. For some reason, this transform falls behind very easily with operations\_behind ranging in 2-10 operations. Any clues? Any more info I must give?

My transform JSONs:

```auto
{
  "id": "1_min_access_logs",
  "authorization": {
    "roles": [
      "superuser"
    ]
  },
  "version": "10.0.0",
  "create_time": 1721693863674,
  "source": {
    "index": [
      "logs-raw_access"
    ],
    "query": {
      "match_all": {}
    }
  },
  "dest": {
    "index": "1_min_access_logs"
  },
  "sync": {
    "time": {
      "field": "@timestamp",
      "delay": "60s"
    }
  },
  "pivot": {
    "group_by": {
      "@timestamp": {
        "date_histogram": {
          "field": "@timestamp",
          "calendar_interval": "1m"
        }
      },
      "user_account": {
        "terms": {
          "field": "user_account"
        }
      },
      "request_method": {
        "terms": {
          "field": "request_method"
        }
      },
      "request_path": {
        "terms": {
          "field": "request_path"
        }
      },
      "response_code": {
        "terms": {
          "field": "response_code"
        }
      }
    },
    "aggregations": {
      "response_time": {
        "avg": {
          "field": "response_time"
        }
      },
      "num_docs": {
        "value_count": {
          "field": "request_id"
        }
      }
    }
  },
  "description": "1_min_access_logs",
  "settings": {}
}

```

```auto
{
  "id": "1_hour_access_logs",
  "authorization": {
    "roles": [
      "superuser"
    ]
  },
  "version": "10.0.0",
  "create_time": 1721694111690,
  "source": {
    "index": [
      "1_min_access_logs"
    ],
    "query": {
      "match_all": {}
    }
  },
  "dest": {
    "index": "1_hour_access_logs"
  },
  "sync": {
    "time": {
      "field": "@timestamp",
      "delay": "120s"
    }
  },
  "pivot": {
    "group_by": {
      "@timestamp": {
        "date_histogram": {
          "field": "@timestamp",
          "calendar_interval": "1h"
        }
      },
      "user_account": {
        "terms": {
          "field": "user_account"
        }
      },
      "request_method": {
        "terms": {
          "field": "request_method"
        }
      },
      "request_path": {
        "terms": {
          "field": "request_path"
        }
      },
      "response_code": {
        "terms": {
          "field": "response_code"
        }
      }
    },
    "aggregations": {
      "num_docs": {
        "sum": {
          "field": "num_docs"
        }
      },
      "response_time_average": {
        "avg": {
          "field": "response_time_average"
        }
      }
    }
  },
  "description": "1_hour_access_logs",
  "settings": {}
}

```

---

<div class="post-metadata">

**Author:** ![Liam\_Young](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/liam_young/32/131373_2.png) [@Liam\_Young](https://discuss.elastic.co/u/Liam_Young)\
**Post date:** [July 23, 2024, 10:39pm UTC](https://discuss.elastic.co/t/transforms-keep-falling-behind-not-processing-any-documents/363596/2 "2024-07-23T22:39:19Z")

</div>

Bump...

---

<div class="post-metadata">

**Author:** ![Liam\_Young](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/liam_young/32/131373_2.png) [@Liam\_Young](https://discuss.elastic.co/u/Liam_Young)\
**Post date:** [July 24, 2024, 9:20pm UTC](https://discuss.elastic.co/t/transforms-keep-falling-behind-not-processing-any-documents/363596/3 "2024-07-24T21:20:19Z")

</div>

Bump...