There are limits to the number of docs per shard of 2 billion, which is a hard lucene limit.
However when doing aggs over an increasing amount of data things will slow down, how slow will depend on what your queries are doing.
Also the more you load into ES the more resources it'll use, which is when scaling horizontally is good as you have more resources to use 
Why are you force refreshing?