I use tranforms to create an aggregated index. The setup are :
- 1 live index with live datas injections. Currently this index receive between 5 and 10 Millions of documents per hour and this will be growth in the future.
- 1 aggregated index, datas comes from 12 Transforms in continuous mode. Transforms are setup to group by date histogram by hour (for making an hourly index) and 5 to 10 other group for split datas for query it. Every transforms calculate 1 to 3 values (count or sum).
Cluster is make with 3 hosts with 12 cpu, 32 Gb of ram and 2Tb of disk per node.
So my problem is than some transforms accumulate delay since the live index and it's a big problem. I have tried to :
- set "max_page_search_size" to 10 000
- set frequency to 30s
But no really change... It have a tricks or a tweak to speed up transforms ?