We plan out scaling up to bigger server not more servers (scaling out). We will only have one server in deployment. We were trying to speed up query performance using aggregation due to the indices being to large causing query performance to be slow. We thought by scaling out the number of instances of Elasticsearch we could spread the load across multiple instances which would improve the performance of the aggregation.
That's correct. Elasticsearch generally scales better horizontally than vertically.
Basically I am seeing the same number of indices with the same document count that return the same results in each of the nodes when query it. I not sure that this should be correct if it is then I fine with it, but I am thinking that it is duplicating the same data in 3 nodes. Is there a way to verify this conclusion?
Cluster state is global. Any node can be queried regardless of where the data is actually stored. The behavior you see is normal. Don't worry about it.