Looks like the join attribute do not have the EagerGlobalOrdinals flag while Keyword and Text have it.
The flag is import to tune the performance. I can manually build json request to change the join field mapping. However, I still want to know how to do it using the auto mapping attributes.