So, as there are no other ways to tell ES to pick Policies.
I had to create a bash script in the pipeline, which would create policies and index templates one after the other using api(curl) on the master node.
Hope ES would come up with features where it would be easy to pass array of policy & template path.