Continuous transform missing some documents, but they show in /_preview

Elastic Stack Kibana
6

Using the _update API is not supported for continuous transforms, it may or may not work. In a nutshell a continuous transform works in 2 steps:

  1. find the changes
  2. recalculate the changed buckets

If you use _update its not visible for step 1. However if you change the bucket by another non _update change almost at the same time, you are lucky.

For seeing what transform does behind the scenes, you can use:

PUT /_cluster/settings
{
   "transient": {
      "logger.org.elasticsearch.xpack.transform.transforms": "trace"
   }
}

to see the queries it runs.

Sporadic messages shouldn't be a problem, however logstash has a batch delay. I still think the docs are missed due to ingest delays.

Again, moving the timestamp to ingest could help to get it closer to indexing. If you want to keep the logstash one, you can use another timestamp field (might also be helpful for debugging, e.g. by aggregating the difference between the two).