# Trying to create tranform job which would not go through all documents

**URL:** <https://discuss.elastic.co/t/trying-to-create-tranform-job-which-would-not-go-through-all-documents/328905>\
**Category:** Kibana\
**Tags:** transforms\
**Created:** [March 30, 2023, 8:50am UTC](https://discuss.elastic.co/t/trying-to-create-tranform-job-which-would-not-go-through-all-documents/328905 "2023-03-30T08:50:03Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kiril\_Karamanolev](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kiril_karamanolev/32/119195_2.png) [@Kiril\_Karamanolev](https://discuss.elastic.co/u/Kiril_Karamanolev)\
**Post date:** [March 30, 2023, 8:50am UTC](https://discuss.elastic.co/t/trying-to-create-tranform-job-which-would-not-go-through-all-documents/328905/1 "2023-03-30T08:50:03Z")

</div>

Hello,

Started using transform but I am struggling to find how to look at only recent documents not from the beginning (because I have 1-year historical data)

The JSON of the job is:

```auto
{
  "id": "ops_authrate_1m",
  "authorization": {
    "roles": [
      "machine_learning_admin",
      "kibana_admin",
      "superuser",
      "rollup_admin",
      "Admin",
      "monitoring_user",
      "kibana_system",
      "metricbeat_internal"
    ]
  },
  "version": "8.4.1",
  "create_time": 1677688246676,
  "source": {
    "index": [
      "opsrptlog-prod-*"
    ],
    "query": {
      "bool": {
        "should": [
          {
            "match_phrase": {
              "ops_event.transaction_type.keyword": "Card_Authorize"
            }
          }
        ],
        "minimum_should_match": 1
      }
    }
  },
  "dest": {
    "index": "prod-authrate-opsrpt"
  },
  "sync": {
    "time": {
      "field": "@timestamp",
      "delay": "60s"
    }
  },
  "pivot": {
    "group_by": {
      "ops_event.company_name": {
        "terms": {
          "field": "ops_event.company_name.keyword"
        }
      },
      "@timestamp": {
        "date_histogram": {
          "field": "@timestamp",
          "calendar_interval": "1m"
        }
      }
    },
    "aggregations": {
      "total_events": {
        "value_count": {
          "field": "ops_event.status.keyword"
        }
      },
      "processed": {
        "filter": {
          "term": {
            "ops_event.status.keyword": "Processed"
          }
        },
        "aggs": {
          "all_processed": {
            "value_count": {
              "field": "ops_event.status.keyword"
            }
          }
        }
      },
      "percentage": {
        "bucket_script": {
          "buckets_path": {
            "success": "processed>all_processed",
            "total": "total_events"
          },
          "script": "params.success / params.total * 100"
        }
      }
    }
  },
  "settings": {
    "docs_per_second": 20000
  },
  "retention_policy": {
    "time": {
      "field": "@timestamp",
      "max_age": "32d"
    }
  }
}

```

I guess I am looking for some range which to specify looking for example last 1 day.

Thank you

---

<div class="post-metadata">

**Author:** ![Hendrik\_Muhs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendrik_muhs/32/25802_2.png) [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)\
**Post date:** [March 30, 2023, 9:44am UTC](https://discuss.elastic.co/t/trying-to-create-tranform-job-which-would-not-go-through-all-documents/328905/2 "2023-03-30T09:44:30Z")

</div>

Best practice is to add a range query with an _absolute_ start date, e.g.

```auto
"query" : {
          "range": {
            "@timestamp": {
              "gte": "2023-01-01T00:00:00"
            }
          }
        }

```

> [@Kiril\_Karamanolev](#):
>
> I guess I am looking for some range which to specify looking for example last 1 day.

Using date math(e.g. `now-1d`) can cause performance issues, because date math causes cache misses. Use an _absolute_ start date instead. Transform will take care of the rest, [checkpointing](https://www.elastic.co/guide/en/elasticsearch/reference/current/transform-checkpoints.html) optimizes the queries for you.

---

<div class="post-metadata">

**Author:** ![Kiril\_Karamanolev](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kiril_karamanolev/32/119195_2.png) [@Kiril\_Karamanolev](https://discuss.elastic.co/u/Kiril_Karamanolev)\
**Post date:** [March 30, 2023, 1:01pm UTC](https://discuss.elastic.co/t/trying-to-create-tranform-job-which-would-not-go-through-all-documents/328905/3 "2023-03-30T13:01:42Z")

</div>

Thanks for the response.

I have created this to put in dev tools:

```auto
PUT _transform/test1
{
  "source": {
    "index": "opsrptlog-prod-*",
    "query": {
      "range": {
        "@timestamp": {
          "gte": "2023-03-01T00:00:00"
        }
      },
      "bool": {
        "should": [
          {
            "match_phrase": {
              "ops_event.transaction_type.keyword": "Card_Authorize"
            }
          }
        ],
        "minimum_should_match": 1
      }
    }
  },
  "pivot": {
    "group_by": {
      "ops_event.company_name": {
        "terms": {
          "field": "ops_event.company_name.keyword"
        }
      },
      "@timestamp": {
        "date_histogram": {
          "field": "@timestamp",
          "calendar_interval": "1m"
        }
      }
    },
    "aggregations": {
      "total_events": {
        "value_count": {
          "field": "ops_event.status.keyword"
        }
      },
      "processed": {
        "filter": {
          "term": {
            "ops_event.status.keyword": "Processed"
          }
        },
        "aggs": {
          "all_processed": {
            "value_count": {
              "field": "ops_event.status.keyword"
            }
          }
        }
      },
      "percentage": {
        "bucket_script": {
          "buckets_path": {
            "success": "processed>all_processed",
            "total": "total_events"
          },
          "script": "params.success / params.total * 100"
        }
      }
    }
  },
  "description": "Test description",
  "dest": {
    "index": "test1"
  },
  "frequency": "5m",
  "retention_policy": {
    "time": {
      "field": "@timestamp",
      "max_age": "32d"
    }
  }
}

```

And when I try to push it I get:

```auto
{
  "error": {
    "root_cause": [
      {
        "type": "parsing_exception",
        "reason": "[range] malformed query, expected [END_OBJECT] but found [FIELD_NAME]",
        "line": 1,
        "col": 55
      }
    ],
    "type": "x_content_parse_exception",
    "reason": "[20:5] [data_frame_transform_config] failed to parse field [source]",
    "caused_by": {
      "type": "x_content_parse_exception",
      "reason": "[20:5] [data_frame_config_source] failed to parse field [query]",
      "caused_by": {
        "type": "parsing_exception",
        "reason": "[range] malformed query, expected [END_OBJECT] but found [FIELD_NAME]",
        "line": 1,
        "col": 55
      }
    }
  },
  "status": 400
}

```

Thank you for looking into this.

---

<div class="post-metadata">

**Author:** ![Hendrik\_Muhs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendrik_muhs/32/25802_2.png) [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)\
**Post date:** [March 30, 2023, 3:14pm UTC](https://discuss.elastic.co/t/trying-to-create-tranform-job-which-would-not-go-through-all-documents/328905/4 "2023-03-30T15:14:14Z")

</div>

as you already have the other query clause, you have to combine them. Both can go into `filter`, because your _should_ is really a _must_. `filter` is the better `must` if you don't need scoring:

```auto
      "bool": {
        "filter": [
          {
            "match_phrase": {
              "ops_event.transaction_type.keyword": "Card_Authorize"
            }
          },
          {
            "range": {
              "@timestamp": {
                "gte": "2023-03-01T00:00:00"
              }
           }
         }
        ]
      }

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 27, 2023, 3:14pm UTC](https://discuss.elastic.co/t/trying-to-create-tranform-job-which-would-not-go-through-all-documents/328905/5 "2023-04-27T15:14:30Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.