# Continuous transform missing some documents, but they show in /\_preview
**URL:** https://discuss.elastic.co/t/continuous-transform-missing-some-documents-but-they-show-in-preview/237614
**Category:** Kibana
**Created:** [June 18, 2020, 10:18am UTC](https://discuss.elastic.co/t/continuous-transform-missing-some-documents-but-they-show-in-preview/237614 "2020-06-18T10:18:07Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@Mark\_Duncan](https://discuss.elastic.co/u/Mark_Duncan)
#### Post date: [June 18, 2020, 10:18am UTC](https://discuss.elastic.co/t/continuous-transform-missing-some-documents-but-they-show-in-preview/237614/1 "2020-06-18T10:18:07Z")
I received 2 new documents earlier within a short time frame.
**Neither of these msgs were updated in my transform output that is running in continuous mode.**
However, if I run the same transform in the console with /\_preview the expected results are shown.
I set delay to 5s yesterday in order to counter any potential ingest delay, but I don't think as long as 5s should be required if I am using the @timestamp field, correct?
I checked the live transform's stats, and there was a checkpoint c.5 seconds after these msgs. I didn't have any other msgs at that time. So it looks like the transform ran...
I need help - why would the msgs get missed in the transform output, when they do show up in /\_preview?
doc 1
@timestamp Jun 18, 2020 @ 07:56:18.383
message {"recv\_timestamp":"2020-06-18T06:56:17.362975577Z","exchange":"dflgateway","instrument":"BTC-USDT","client\_oid":"e7203f297b1044dbbc03f50c0a089383","order\_id":"5092674538389505","side":"sell","price":9450,"quantity":0.0012,"limit\_or\_market":"limit","order\_type":"0","status":"filled","state":2,"notional":0,"filled\_notional":11.34,"filled\_size":0.0012,"last\_fill\_px":9450,"last\_fill\_qty":0.0012,"last\_fill\_time":"2020-06-18T06:56:17.305000000Z","margin\_trading":2,"exchange\_timestamp":"2020-06-18T06:56:17.305000000Z","created\_at":"2020-06-18T01:34:06.546000000Z","extra\_fields":{"identity":"3548","logon\_id":"master"}}
doc 2
@timestamp Jun 18, 2020 @ 07:56:18.383
message {"recv\_timestamp":"2020-06-18T06:56:17.463456605Z","exchange":"dflgateway","instrument":"BTC-USDT","client\_oid":"1b5fdec92b714cfdbc17afc73e3124be","order\_id":"5093941407147008","side":"buy","price":9425,"quantity":0.0012,"limit\_or\_market":"limit","order\_type":"0","status":"open","state":0,"notional":0,"filled\_notional":0,"filled\_size":0,"last\_fill\_px":0,"last\_fill\_qty":0,"last\_fill\_time":"1970-01-01T00:00:00.000000000Z","margin\_trading":2,"exchange\_timestamp":"2020-06-18T06:56:17.429000000Z","created\_at":"2020-06-18T06:56:17.429000000Z","extra\_fields":{"identity":"3548","logon\_id":"master"}}
And here is my transform script:
> {
> "id": "dflg\_margin\_limit\_order\_status\_v7",
> "source": {
> "index": [
> "dflg-okex-account-orders\*"
> ],
> "query": {
> "bool": {
> "filter": [
> {
> "bool": {
> "should": [
> {
> "match": {
> "margin\_trading": 2
> }
> }
> ],
> "minimum\_should\_match": 1
> }
> },
> {
> "bool": {
> "filter": [
> {
> "bool": {
> "should": [
> {
> "match\_phrase": {
> "limit\_or\_market": "limit"
> }
> }
> ],
> "minimum\_should\_match": 1
> }
> }
> ]
> }
> }
> ]
> }
> }
> },
> "dest": {
> "index": "dflg\_margin\_limit\_order\_status"
> },
> "frequency": "1s",
> "sync": {
> "time": {
> "field": "@timestamp",
> "delay": "5s"
> }
> },
> "pivot": {
> "group\_by": {
> "order\_id": {
> "terms": {
> "field": "order\_id.keyword"
> }
> },
> "side": {
> "terms": {
> "field": "side.keyword"
> }
> },
> "datatype": {
> "terms": {
> "field": "datatype.keyword"
> }
> },
> "extra\_fields.identity": {
> "terms": {
> "field": "extra\_fields.identity.keyword"
> }
> },
> "instrument": {
> "terms": {
> "field": "instrument.keyword"
> }
> },
> "quantity": {
> "histogram": {
> "field": "quantity",
> "interval": "0.00000001"
> }
> },
> "price": {
> "histogram": {
> "field": "price",
> "interval": "0.00000000001"
> }
> }
> },
> "aggregations": {
> "filled\_notional.max": {
> "max": {
> "field": "filled\_notional"
> }
> },
> "created\_at": {
> "max": {
> "field": "created\_at"
> }
> },
> "recv\_timestamp": {
> "max": {
> "field": "recv\_timestamp"
> }
> },
> "filled\_size.max": {
> "max": {
> "field": "filled\_size"
> }
> },
> "last\_status": {
> "scripted\_metric": {
> "init\_script": "state.filled\_size\_latest = 0.0; state.last\_status = ''; state.recv\_timestamp\_latest = 0L",
> "map\_script": "\n def current\_filled\_size = doc['filled\_size'].value;\n def current\_recv\_timestamp = doc.recv\_timestamp.value.toInstant().toEpochMilli();\n if (current\_filled\_size \> state.filled\_size\_latest)\n {\n state.filled\_size\_latest = current\_filled\_size;\n state.last\_status = params['\_source']['status'];\n state.recv\_timestamp\_latest = current\_recv\_timestamp;\n }\n else if (current\_recv\_timestamp \> state.recv\_timestamp\_latest)\n {\n state.filled\_size\_latest = current\_filled\_size;\n state.last\_status = params['\_source']['status'];\n state.recv\_timestamp\_latest = current\_recv\_timestamp;\n }\n else if (params['\_source']['status'] == 'cancelled')\n {\n state.last\_status = params['\_source']['status'];\n }\n ",
> "combine\_script": "return state",
> "reduce\_script": "\n def last\_status = '';\n def filled\_size\_latest = 0L;\n def recv\_timestamp\_latest = 0.0;\n for (s in states) \n {\n if (s.filled\_size\_latest \> filled\_size\_latest)\n {\n filled\_size\_latest = s.filled\_size\_latest; last\_status = s.last\_status;\n }\n else if (s.recv\_timestamp\_latest \> recv\_timestamp\_latest)\n {\n recv\_timestamp\_latest = s.recv\_timestamp\_latest; last\_status = s.last\_status;\n }\n }\n return last\_status\n "
> }
> }
> }
> },
> "version": "7.5.2",
> "create\_time": 1592406746768
> }
---
### Author:  [@Mark\_Duncan](https://discuss.elastic.co/u/Mark_Duncan)
#### Post date: [June 18, 2020, 2:32pm UTC](https://discuss.elastic.co/t/continuous-transform-missing-some-documents-but-they-show-in-preview/237614/2 "2020-06-18T14:32:35Z")
Perhaps I can summarise my question here - my continuous transform is live; it recorded a checkpoint 5s after new docs arrived, but failed to update the transform output at that time with the new documents.
How do I solve this?
The expected results - including the new documents - are shown to me if I call /\_preview in the console.
---
### Author:  [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)
#### Post date: [June 18, 2020, 7:22pm UTC](https://discuss.elastic.co/t/continuous-transform-missing-some-documents-but-they-show-in-preview/237614/3 "2020-06-18T19:22:07Z")
When a checkpoint is created, lets say at time `x`, it creates an upper bound `x-delay` and than requires that any new data is after `x-delay`, if you insert/update old data, a continuous transform is not able to see the change. This is how continuous transform scale. The [docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/transform-checkpoints.html) go a bit more into detail how it works.
Having that said, we need to find out why this did not work for you:
- How is `@timestamp` created?
- What kind of ingest processing is running upfront?
By setting a delay of `5s` you assume a quite low ingest processing delay. Note that `1s` is already the minimum if you use the default `refresh_interval`. The default delay is `60s`.
- Do you use the default or did you changed the `refresh_interval` of the source index?
The reason why `_preview` is working is simple, it creates the preview on the full data set. A new transform would also see the 2 documents you missed in the continuous transform.
---
### Author:  [@Mark\_Duncan](https://discuss.elastic.co/u/Mark_Duncan)
#### Post date: [June 19, 2020, 10:33am UTC](https://discuss.elastic.co/t/continuous-transform-missing-some-documents-but-they-show-in-preview/237614/4 "2020-06-19T10:33:06Z")
Thanks @Hendrik_Muhs
@timestamp is the timestamp added by logstash as some metadata. So I think it ought to be (close to) the time the document is indexed by Elasticsearch.
Logstash upfront.
I had set to 5s in an attempt to solve my issue. I'd certainly like to reduce - and have the destination index updated as rapidly as possible.
Msgs to the source index could be quite sporadic - I may not have a msg for an hour, for instance. When a new msg/document is received, I'd like it to appear in the transform destination index ASAP.
I have not changed the `refresh_interval` of the source index.
Thanks for your help
---
### Author:  [@Mark\_Duncan](https://discuss.elastic.co/u/Mark_Duncan)
#### Post date: [June 24, 2020, 12:33pm UTC](https://discuss.elastic.co/t/continuous-transform-missing-some-documents-but-they-show-in-preview/237614/5 "2020-06-24T12:33:40Z")
Hi @Hendrik_Muhs
Any thoughts on why the transform would run but miss some docs?
I think I've seen that updates made via the /\_update API need the transform to be stopped and restarted to fully take effect.
That being said, I've still seen 1 doc missed since I stopped and restarted the transform.
---
### Author:  [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)
#### Post date: [June 24, 2020, 1:03pm UTC](https://discuss.elastic.co/t/continuous-transform-missing-some-documents-but-they-show-in-preview/237614/6 "2020-06-24T13:03:15Z")
Using the `_update` API is not supported for continuous transforms, it may or may not work. In a nutshell a continuous transform works in 2 steps:
1. find the changes
2. recalculate the changed buckets
If you use `_update` its not visible for step 1. However if you change the bucket by another non `_update` change almost at the same time, you are lucky.
For seeing what transform does behind the scenes, you can use:
```auto
PUT /_cluster/settings
{
"transient": {
"logger.org.elasticsearch.xpack.transform.transforms": "trace"
}
}
```
to see the queries it runs.
Sporadic messages shouldn't be a problem, however logstash has a batch [delay](https://www.elastic.co/guide/en/logstash/current/logstash-settings-file.html). I still think the docs are missed due to ingest delays.
Again, moving the [timestamp to ingest](https://discuss.elastic.co/t/dec-12th-2018-en-elasticsearch-automatically-adding-a-timestamp-to-documents/159314) could help to get it closer to indexing. If you want to keep the logstash one, you can use another timestamp field (might also be helpful for debugging, e.g. by aggregating the difference between the two).
---
### Author:  [@mykael](https://discuss.elastic.co/u/mykael)
#### Post date: [June 25, 2020, 5:35am UTC](https://discuss.elastic.co/t/continuous-transform-missing-some-documents-but-they-show-in-preview/237614/7 "2020-06-25T05:35:46Z")
Add the following filter in logstash and it'll tag each record with the date the record was processed by logstash. You'll need to update your index template to say 'ingest\_epoch' is a date, because it'll default to making it a long.
```
ruby {
code => 'event.set("ingest_epoch", (Time.now.to_f * 1000).to_i)'
}
```
Then in your continuous transform, use ingest\_epoch as your sync field with a delay of, say, 60 seconds. That will ensure the transform processes all records added to the index (about a minute after they arrive), regardless of the time period in @timestamp. Then use a group\_by on a date histogram derived from the @timestamp to get it to overwrite the records for each time slot...
---
### Author:  [@Mark\_Duncan](https://discuss.elastic.co/u/Mark_Duncan)
#### Post date: [July 3, 2020, 3:01pm UTC](https://discuss.elastic.co/t/continuous-transform-missing-some-documents-but-they-show-in-preview/237614/8 "2020-07-03T15:01:16Z")
thanks @Hendrik_Muhs I added the ingest\_timestamp and been good ever since. thank you!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 31, 2020, 3:01pm UTC](https://discuss.elastic.co/t/continuous-transform-missing-some-documents-but-they-show-in-preview/237614/9 "2020-07-31T15:01:25Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.