Transform kills the whole cluster

Elastic Stack Elasticsearch
1

Hi I'm using elastic cloud (cloud.elastic.co)

I'm not sure if that's the correct category, but I want to run a transform for the past 6 month, and it's killing the cpu. the whole cluster becomes unresponsive (queries, kibana ..)

is there any option to tell the transform to run "slowly"..?
image

2

Unfortunately not yet, throttling will be part of 7.8, check the docs_per_second setting in the docs.

I can not give you a date when 7.8 will be available, but it should not be far off.

3

ok, i'll run it at night for now. thanks you guys for your hard work

5

It seems to me that your cluster has some general performance problems or is simply under-powered with this task.

To further analyze this problem it seems better to ask support.

Logging queries that transform does, is a good approach. You can take the query from the log and execute it in dev console and see how fast it performs there. I guess it will be slow, too.

Transform can only go as fast as aggregations + bulk indexing.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.