# Transforms updates fields from data that exists from before even a filter condition was not met

**URL:** <https://discuss.elastic.co/t/transforms-updates-fields-from-data-that-exists-from-before-even-a-filter-condition-was-not-met/256020>\
**Category:** Elasticsearch\
**Tags:** transforms\
**Created:** [November 19, 2020, 4:50pm UTC](https://discuss.elastic.co/t/transforms-updates-fields-from-data-that-exists-from-before-even-a-filter-condition-was-not-met/256020 "2020-11-19T16:50:35Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![syunusic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/syunusic/32/4131_2.png) [@syunusic](https://discuss.elastic.co/u/syunusic)\
**Post date:** [November 19, 2020, 4:50pm UTC](https://discuss.elastic.co/t/transforms-updates-fields-from-data-that-exists-from-before-even-a-filter-condition-was-not-met/256020/1 "2020-11-19T16:50:35Z")

</div>

I have an index (client\_index) that has three fields: @timestamp, user, ip. I have a transform like this:

```auto
{
  "id": "my_transform",
  "source": {
    "index": [
      "client_index"
    ],
    "query": {
      "match_all": {}
    }
  },
  "dest": {
    "index": "client_index_transformed"
  },
  "sync": {
    "time": {
      "field": "@timestamp",
      "delay": "60s"
    }
  },
  "pivot": {
    "group_by": {
      "user": {
        "terms": {
          "field": "user"
        }
      }
    },
    "aggregations": {
      "@timestamp.max": {
        "max": {
          "field": "@timestamp"
        }
      },
      "srcip.filter": {
        "filter": {
          "range": {
            "srcip": {
              "gt": "192.168.140.1",
              "lt": "192.168.143.254"
            }
          }
        }
      }
    }
  },
  "description": "hi",
  "settings": {},
  "version": "7.9.2",
  "create_time": 1604933401081
}

```

And I detect a problem...  
If a user has at some point (let's say one month ago) has IP 192.168.140.2, the transform will put it into the client\_index\_transformed index.. which is good...

The problem is that today the user has IP 192.168.240.3 shouldn't update the client\_index\_transformed index with the timestamp.. but it does..

So, If I query the client\_index\_transformed index for that user, the max @timestamp will be today and not one month ago... why is that and how to avoid it?

Thanks!

---

<div class="post-metadata">

**Author:** ![przemekwitek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/przemekwitek/32/79526_2.png) [@przemekwitek](https://discuss.elastic.co/u/przemekwitek)\
**Post date:** [November 23, 2020, 12:53pm UTC](https://discuss.elastic.co/t/transforms-updates-fields-from-data-that-exists-from-before-even-a-filter-condition-was-not-met/256020/2 "2020-11-23T12:53:42Z")

</div>

Hi,  
In the above transform snippet, you specified 2 aggs ("@timestamp.max", "srcip.filter") but they are not related to each other, i.e. the latter ("srcip.filter") does not restrict the set of documents to perform the former ("@timestamp.max"). In fact, these 2 aggs are performed independently, that's why you see the updated row in destination index for the user in interest.

You can try nesting the "@timestamp.max" aggregation inside "srcip.filter" this way:

```auto
    "aggregations": {
      "srcip.filter": {
        "filter": {
          "range": {
            "srcip": {
              "gt": "192.168.140.1",
              "lt": "192.168.143.254"
            }
          }
        },
        "aggs": {
          "@timestamp.max": {
            "max": {
              "field": "@timestamp"
            }
          }
        }
      }
    }  

```

and see if this produces the result you expect.

---

<div class="post-metadata">

**Author:** ![syunusic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/syunusic/32/4131_2.png) [@syunusic](https://discuss.elastic.co/u/syunusic)\
**Post date:** [November 24, 2020, 6:08pm UTC](https://discuss.elastic.co/t/transforms-updates-fields-from-data-that-exists-from-before-even-a-filter-condition-was-not-met/256020/3 "2020-11-24T18:08:24Z")

</div>

It worked. Thanks!  
So, basically, what I had was an "OR" instead of and "AND", right?  
I think the only way to do it is editing the json config in kibana 7.9.2, right?

---

<div class="post-metadata">

**Author:** ![przemekwitek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/przemekwitek/32/79526_2.png) [@przemekwitek](https://discuss.elastic.co/u/przemekwitek)\
**Post date:** [November 25, 2020, 10:03am UTC](https://discuss.elastic.co/t/transforms-updates-fields-from-data-that-exists-from-before-even-a-filter-condition-was-not-met/256020/4 "2020-11-25T10:03:58Z")

</div>

I'm glad it worked!

> So, basically, what I had was an "OR" instead of and "AND", right?

You can think of aggs specified in a transform as separate (independent) fields you'll get in the destination index. In this sense it definitely is more like "OR" than "AND".

> I think the only way to do it is editing the json config in kibana 7.9.2, right?

I think so. There is a "Edit JSON config" toggle button next to the Aggregations section

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 23, 2020, 10:04am UTC](https://discuss.elastic.co/t/transforms-updates-fields-from-data-that-exists-from-before-even-a-filter-condition-was-not-met/256020/5 "2020-12-23T10:04:00Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.