Unfortunately what you said is already the recommended approach by elastic, it is documented here.
You would need to to create a custom template for every dataset and every namespace, which is something really hard to maintain.
I had the same issue when I started to use the integrations to get some logs and by suggestion of someone from Elastic I opened this issue on Github proposing some changes.
Iif you can wait I would suggest that you keep using Beats instead of Elastic Agent, in my experience Elastic Agent makes it easier to get the data, but make the management of the indices twice or more hard.