# Transform - Continuous mode for more than 1 index?

**URL:** <https://discuss.elastic.co/t/transform-continuous-mode-for-more-than-1-index/239244>\
**Category:** Elasticsearch\
**Created:** [June 30, 2020, 7:39am UTC](https://discuss.elastic.co/t/transform-continuous-mode-for-more-than-1-index/239244 "2020-06-30T07:39:43Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jason\_Neo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jason_neo/32/71342_2.png) [@Jason\_Neo](https://discuss.elastic.co/u/Jason_Neo)\
**Post date:** [June 30, 2020, 7:39am UTC](https://discuss.elastic.co/t/transform-continuous-mode-for-more-than-1-index/239244/1 "2020-06-30T07:39:43Z")

</div>

Hi all.

I have been searching the web and elastic forum looking for solution, but it doesn't seem like people are having problem with the Transform function.

I'm using Transform to create summary tables to optimize queries. It's working if i were to create a new transformation. However, everyday a newly index with the same index pattern will be added to elasticsearch. I knowing that there is the capability to continuously perform transformation by checks for changes to source indices continuously . How should i configure to make it working?

For e.g.  
The following are the indices, on each day, a new index will be added.  
snort-2020-06-01  
snort 2020-06-02  
snort 2020-06-03  
....  
snort 2020-06-30 (newly added)

When doing transform, i chose the index pattern "snort\*".

I want the "continuous transform function" to pick up new index pattern and update the transformed index.

Appreciate much!

---

<div class="post-metadata">

**Author:** ![Jason\_Neo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jason_neo/32/71342_2.png) [@Jason\_Neo](https://discuss.elastic.co/u/Jason_Neo)\
**Post date:** [July 1, 2020, 3:18am UTC](https://discuss.elastic.co/t/transform-continuous-mode-for-more-than-1-index/239244/2 "2020-07-01T03:18:03Z")

</div>

![image](https://us1.discourse-cdn.com/elastic/original/3X/f/f/ffa9f39a2aa765e78a5b2970fbac98839dc4c057.png)

refering to to the above image, source index is **snort** \*

---

<div class="post-metadata">

**Author:** ![Hendrik\_Muhs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendrik_muhs/32/25802_2.png) [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)\
**Post date:** [July 1, 2020, 7:06am UTC](https://discuss.elastic.co/t/transform-continuous-mode-for-more-than-1-index/239244/3 "2020-07-01T07:06:23Z")

</div>

Your approach looks fine to me, you can use index patterns with wildcards as source of the transform, an alias that points to multiple indices should work as well.

If you are using `date_histogram` in your `group_by` it's advised to use at least `7.7`, which introduced an optimization for this case.

If you need specific help, please post your job configuration or at least the parts you have questions.

---

<div class="post-metadata">

**Author:** ![Jason\_Neo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jason_neo/32/71342_2.png) [@Jason\_Neo](https://discuss.elastic.co/u/Jason_Neo)\
**Post date:** [July 1, 2020, 2:08pm UTC](https://discuss.elastic.co/t/transform-continuous-mode-for-more-than-1-index/239244/4 "2020-07-01T14:08:56Z")

</div>

Hey @Hendrik_Muhs

Thanks for your reply, but my new indices are not being picked up by the transformed\_snort job. Currently i'm just using one node for elasticsearch, but i suppose it will not affect the transformation job?

The following is my job configuration

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/4/b40a63c8bb33fa7874e9d3ad00189ecbd9f741e3.png)

In part 2 of the configuration, I suppose the Date field '@timestamp' refers to the timestamp in the index pattern(snort\*)?  
As per my requirement, I do not have any @timestamp in my transformed\_snort except @timestamp.max and @timestamp.min for aggregations.

---

<div class="post-metadata">

**Author:** ![Hendrik\_Muhs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendrik_muhs/32/25802_2.png) [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)\
**Post date:** [July 1, 2020, 2:50pm UTC](https://discuss.elastic.co/t/transform-continuous-mode-for-more-than-1-index/239244/5 "2020-07-01T14:50:56Z")

</div>

When using the kibana UI you are using kibana index patterns. I am not sure, why its not picked up. Can you check `source` in dev console:

```auto
GET _transform/{name}

```

Maybe the kibana index pattern is not setup correctly, but it looks ok to me. How do you know its not picking up new indices? Does visualizations work using the same pattern?

For a continuous transform you specify the timestamp field in the source index, the suggested `@timestamp` looks ok to me.

---

<div class="post-metadata">

**Author:** ![Jason\_Neo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jason_neo/32/71342_2.png) [@Jason\_Neo](https://discuss.elastic.co/u/Jason_Neo)\
**Post date:** [July 1, 2020, 4:17pm UTC](https://discuss.elastic.co/t/transform-continuous-mode-for-more-than-1-index/239244/6 "2020-07-01T16:17:36Z")

</div>

> {  
> "count" : 1,  
> "transforms" : [  
> {  
> "id" : "transformed\_snort",  
> "source" : {  
> "index" : [  
> "snort\*"  
> ],  
> "query" : {  
> "match\_all" : { }  
> }  
> },  
> "dest" : {  
> "index" : "transformed\_snort"  
> },  
> "sync" : {  
> "time" : {  
> "field" : "@timestamp",  
> "delay" : "60s"  
> }  
> },  
> "pivot" : {  
> "group\_by" : {  
> "ip.dst" : {  
> "terms" : {  
> "field" : "ip.dst"  
> }  
> },  
> "ip.src" : {  
> "terms" : {  
> "field" : "ip.src"  
> }  
> },  
> "port.dst" : {  
> "terms" : {  
> "field" : "port.dst"  
> }  
> },  
> "port.src" : {  
> "terms" : {  
> "field" : "port.src"  
> }  
> },  
> "frame.protocols" : {  
> "terms" : {  
> "field" : "frame.protocols"  
> }  
> }  
> },  
> "aggregations" : {  
> "timestamp\_max" : {  
> "max" : {  
> "field" : "@timestamp"  
> }  
> },  
> "timestamp\_min" : {  
> "min" : {  
> "field" : "@timestamp"  
> }  
> }  
> }  
> },  
> "description" : "transformed\_snort",  
> "version" : "7.7.0",  
> "create\_time" : 1593585286020  
> }  
> ]  
> }

Source seems correct here as well.

Everyday there will be new index created. I found that "doc count" and "size" of the transformed\_snort did not increase, and suspected something when wrong with the continuous transform.

Visualization of the index pattern(snort\*) is working well for me.

---

<div class="post-metadata">

**Author:** ![Hendrik\_Muhs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendrik_muhs/32/25802_2.png) [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)\
**Post date:** [July 1, 2020, 6:12pm UTC](https://discuss.elastic.co/t/transform-continuous-mode-for-more-than-1-index/239244/7 "2020-07-01T18:12:26Z")

</div>

ok, thanks. I do not see any problem in your config. The next step towards debugging your case is to take a look at the [stats](https://www.elastic.co/guide/en/elasticsearch/reference/current/get-transform-stats.html):

```auto
GET _transform/{name}/_stats

```

This is also available in the UI if you click on the little arrow left of the transform name.

---

<div class="post-metadata">

**Author:** ![Jason\_Neo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jason_neo/32/71342_2.png) [@Jason\_Neo](https://discuss.elastic.co/u/Jason_Neo)\
**Post date:** [July 2, 2020, 3:07am UTC](https://discuss.elastic.co/t/transform-continuous-mode-for-more-than-1-index/239244/8 "2020-07-02T03:07:04Z")

</div>

> {  
> "count" : 1,  
> "transforms" : [  
> {  
> "id" : "transformed\_snort",  
> "state" : "started",  
> "node" : {  
> "id" : "2\_u7iCyARIG-RiPdv-PyHg",  
> "name" : "instance-2",  
> "ephemeral\_id" : "A3as6ogqTLSrAee0dXcaNQ",  
> "transport\_address" : "127.0.0.1:9300",  
> "attributes" : { }  
> },  
> "stats" : {  
> "pages\_processed" : 3130,  
> "documents\_processed" : 11043063,  
> "documents\_indexed" : 1563650,  
> "trigger\_count" : 230,  
> "index\_time\_in\_ms" : 99674,  
> "index\_total" : 3128,  
> "index\_failures" : 0,  
> "search\_time\_in\_ms" : 12581507,  
> "search\_total" : 3130,  
> "search\_failures" : 0,  
> "processing\_time\_in\_ms" : 21175,  
> "processing\_total" : 3130,  
> "exponential\_avg\_checkpoint\_duration\_ms" : 1.0403123727272727E7,  
> "exponential\_avg\_documents\_indexed" : 1279350.0,  
> "exponential\_avg\_documents\_processed" : 9035233.363636363  
> },  
> "checkpointing" : {  
> "last" : {  
> "checkpoint" : 2,  
> "timestamp\_millis" : 1593611391619,  
> "time\_upper\_bound\_millis" : 1593611331619  
> },  
> "operations\_behind" : 1597137  
> }  
> }  
> ]  
> }

here you go..

---

<div class="post-metadata">

**Author:** ![Hendrik\_Muhs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendrik_muhs/32/25802_2.png) [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)\
**Post date:** [July 2, 2020, 8:17am UTC](https://discuss.elastic.co/t/transform-continuous-mode-for-more-than-1-index/239244/9 "2020-07-02T08:17:09Z")

</div>

Thanks, again I do not see something wrong.

The stats do not contain any error (see the counters for `_failures`), the checkpoint is only 2, which means it created only 1 more, however trigger count is `230`, so it at least checked \> 200 times for updates.

The `time_upper_bound_millis` corresponds to 07/01/2020 @ 1:48pm (UTC). The data you are adding is _not_ before that?

---

<div class="post-metadata">

**Author:** ![Jason\_Neo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jason_neo/32/71342_2.png) [@Jason\_Neo](https://discuss.elastic.co/u/Jason_Neo)\
**Post date:** [July 2, 2020, 8:39am UTC](https://discuss.elastic.co/t/transform-continuous-mode-for-more-than-1-index/239244/10 "2020-07-02T08:39:12Z")

</div>

Thanks @Hendrik_Muhs !

Seems like the root cause is `time_upper_bound_millis`. My data are before this date.

I'm not sure how this `time_upper_bound_millis` is being populated when my data do not have this datetime.

Do you know how this timestamp is added in and how can i force a re-transform (re-index)?

---

<div class="post-metadata">

**Author:** ![Hendrik\_Muhs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendrik_muhs/32/25802_2.png) [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)\
**Post date:** [July 2, 2020, 11:32am UTC](https://discuss.elastic.co/t/transform-continuous-mode-for-more-than-1-index/239244/11 "2020-07-02T11:32:09Z")

</div>

The concept of a continuous transform is to continually increment and process checkpoints as new source data is ingested. The timestamp used for synchronizing `source` and `dest` **must** follow real time, meaning it must be a recent timestamp. To adjust for index delays, e.g. because the timestamp you use runs behind due to processing delays, you can use the `delay` parameter, default `60s`. That causes transform to deduct the delay when querying data, e.g. `lt now-delay`.

If you process historic data, there is no need to use a continuous transform, but you can use a batch transform. Is there a reason you want to process historic data but still use continuous mode?

There is a trick, instead of using the historic `timestamp`, you can add an ingest timestamp while you are feeding in new data, here is [how](https://discuss.elastic.co/t/dec-12th-2018-en-elasticsearch-automatically-adding-a-timestamp-to-documents/159314). You than use the ingest timestamp for `sync`, you can keep your `timestamp_max` and `timestamp_min` as is.

---

<div class="post-metadata">

**Author:** ![Jason\_Neo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jason_neo/32/71342_2.png) [@Jason\_Neo](https://discuss.elastic.co/u/Jason_Neo)\
**Post date:** [July 6, 2020, 1:28am UTC](https://discuss.elastic.co/t/transform-continuous-mode-for-more-than-1-index/239244/12 "2020-07-06T01:28:54Z")

</div>

@Hendrik_Muhs

Thank you for clarifying. I am testing out a solution that maybe deployed in production, so i wanted to mimic it as far as possible but i do not have access to real-time data.

But I think the "trick" by creating a document\_created\_datetime will work for both dev/prod.

A big thank you for your help. 👍

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 3, 2020, 1:28am UTC](https://discuss.elastic.co/t/transform-continuous-mode-for-more-than-1-index/239244/13 "2020-08-03T01:28:56Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.