It is for time-based logs monitoring.
To give you a bit more context, what we have is a large number of existing servers on different local high speed networks on which there are logs generated by applications running on those servers. To give you rough sense of scale, each local network has around at least 1000+ servers each generating 20GB logs per day and we are aiming to keep at least 2 weeks worth.
Because the amount of logs is too big, we thought if we were to set up dedicated infra to run elasticsearch clusters it would be setting up too many new servers, perhaps hundreds so why not re-use the existing servers that we already have to host elasticsearch nodes as the existing servers are powerful but more importantly there are still quite a lot of free space on these servers - that free space when summed up together, we calculated, is more than enough to serve our purpose.
But I am concerned about issues that I might not know of when having too many nodes within a single cluster so was wondering whether it's ok to have a monolithic cluster with 1000 nodes in a local network or whether I should go with multiple smaller clusters inside the same local network.