The question is "why a user would like to see the less relevant results?"
Page 20 is less relevant than page 1.
Do you go to page 20 when you search on Google?
I don't think you do. Why your users would have to do this?
After the fact it's slower: yes. That's one of the price to pay. The other is memory consumption.