# Transformed index is missing data

**URL:** <https://discuss.elastic.co/t/transformed-index-is-missing-data/264725>\
**Category:** Elasticsearch\
**Tags:** transforms\
**Created:** [February 18, 2021, 2:53pm UTC](https://discuss.elastic.co/t/transformed-index-is-missing-data/264725 "2021-02-18T14:53:27Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![kazuki](https://avatars.discourse-cdn.com/v4/letter/k/e9c0ed/32.png) [@kazuki](https://discuss.elastic.co/u/kazuki)\
**Post date:** [February 18, 2021, 2:53pm UTC](https://discuss.elastic.co/t/transformed-index-is-missing-data/264725/1 "2021-02-18T14:53:27Z")

</div>

We have a transform that creates aggregation based on userId and clientID from a source index and write to a destination index. We've noticed the destination index is missing data it should have even after waiting 10+ minutes (sync frequency is set as 1m with sync delay also 1m). We have some data from similar timerange in destination while others are missing. What could the reason for missing data?

Below is transform definition (I renamed index name for sharing)

```auto
{
  "source": {
    "index": "source_index",
    "query": {
      "bool": {
        "must_not": [
          {"term": {"userId": ""}}
        ]
      }
    }
  },
  "dest": {
    "index": "dest_index",
    "pipeline": "add_timestamps_v2"
  },
  "pivot": {
    "group_by": {
      "client_id": {
        "terms": {
          "field": "client_id"
        }
      },
      "user_id_hash": {
        "terms": {
          "field": "userId"
        }
      }
    },
    "aggs": {
      "devices": {
        "terms": {
          "field": "device_id"
        },
        "aggs": {
          "users": {
            "terms": {
              "field": "userId"
            }
          }
        }
      }
      // more aggregations here..
    }
  },
  "frequency": "1m",
  "sync": {
    "time": {
      "field": "updated_at",
      "delay": "60s"
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [February 22, 2021, 3:51am UTC](https://discuss.elastic.co/t/transformed-index-is-missing-data/264725/2 "2021-02-22T03:51:59Z")

</div>

Welcome to our community! 😃

How are you identifying the missing data?

---

<div class="post-metadata">

**Author:** ![kazuki](https://avatars.discourse-cdn.com/v4/letter/k/e9c0ed/32.png) [@kazuki](https://discuss.elastic.co/u/kazuki)\
**Post date:** [February 22, 2021, 10:47pm UTC](https://discuss.elastic.co/t/transformed-index-is-missing-data/264725/3 "2021-02-22T22:47:12Z")

</div>

Hi @warkolm we do have a log in our application for missing data, and I confirmed that data is missing in destination index by searching with client\_id and user\_id

---

<div class="post-metadata">

**Author:** ![Hendrik\_Muhs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendrik_muhs/32/25802_2.png) [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)\
**Post date:** [February 23, 2021, 8:20am UTC](https://discuss.elastic.co/t/transformed-index-is-missing-data/264725/4 "2021-02-23T08:20:53Z")

</div>

> [@kazuki](#):
>
> We've noticed the destination index is missing data it should have even after waiting 10+ minutes (sync frequency is set as 1m with sync delay also 1m).

Which field are you using for `sync` and how is the timestamp created?

`frequency` controls how often transforms looks for new data and/or retries after a failure.  
`sync.delay` compensates ingest delays, meaning it defines how long transform waits for new data to arrive late and/or out of order

I assume your problem could be the setting for `sync.delay` as you stated you waited 10+ minutes, but that does not matter. If a checkpoint is created it takes all the data that is available at that time, if data comes in late, it is not taken into account. It's like missing a flight, while the next ones are all fully booked.

Example: Assume `sync.delay` is set to `1m`. When a checkpoint is created all data between `(old_checkpoint, now() - 1m]` is queried and processed. If data that falls into that range arrives later, it is neither part of this nor the next checkpoint, because for the next checkpoint it is considered too old.

I suggest you investigate whether `sync.delay` is set correctly for your use case. I assume you have to increase it.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 23, 2021, 8:21am UTC](https://discuss.elastic.co/t/transformed-index-is-missing-data/264725/5 "2021-03-23T08:21:15Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.