Why search.max_buckets soft limit is 10000 ? What are the cons of increasing this size limit?

Also, why top_hits window inside aggregation buckets is 100. Does that mean, increasing the window size to 10 times will have a significant toll on performance ?

Not sure but if the size is greater than 100, it might possibly cause error as in this thread

These limits are typically memory-related. Trying to carry too much back in a single response is a way to put servers under memory pressure - especially if your system is dealing with multiple concurrent users.
For large result sets you'll want to break single requests into multiple using something like the composite aggregation and the after parameter to page through results.

Thank you