Hey Folks,
Just wanted to say, thanks to everyone in helping me get though the upgrade from 2.4 -> 5.4.1
It was difficult trying to figuring out all the little details and changes that 5.X introduced but you all helped me though it.
THANKS!
So now I have a lean mean log processing machine
2.4 Indexing only maxing out at ~12K per second
1 Big Cluster
14 Servers (40 CPU's and +128GB ram ) 40TB EMC Storage
* 20 Indexes a day ( 16 deleted after 5 days 4 kept for 20 days )
* >4000 shards (30K at one point)
* 70 Data Nodes
* 3 masters
* 3 clients
* 3 Kafka Nodes
* 6 indexing Logstash servers
* 150 Servers forwarding logs
5.4.1 Not sure of the max yet, but processing 30K messages per second
5 Clusters ( Separated by Business Unit 4 in NY 1 in JXN,Ms)
14 Servers + 4 new VM in remote data center
* 10 Indexes a day ( 16 deleted after 5 days 4 kept for 20 days )
* 4 shards per index (about 8 shards a day per cluster )
* 4 Data Nodes ( per cluster )
* 3 masters ( per cluster)
* 1 clients (per cluster )
* 3 Kafka Nodes ( TOTAL)
* 1 Tribe Node ( I am aware of it being deprecated as of 5.3 but the Cross Cluster search is a new concept to me)
* 5 indexing Logstash servers
* 150 Servers forwarding logs