We've been playing with watcher in our dev environment and were just about to deploy it on our main, apx 15 node cluster that hosts 10 TB of data. I just now saw "If you have a larger cluster, we recommend running Watcher on a separate monitoring cluster. If you’re using a separate cluster for Marvel, you can install Watcher on the nodes you’re using to store your Marvel data" here.
Can you give us some more detail about why this is recommended and the pros/cons of each deployment option (separate cluster vs. on the main cluster)? We're currently running elasticsearch 1.7.2, on IAAS VMs in Azure, and plan to upgrade to the latest version (or at least 2.something) within the next few months.
Also, specifically, we plan to use the index action with watches to load data from the watch payload into an index. If we have watcher running on a separate cluster, can we use the index action to load the data from the watch payload back into an index on the external cluster we're watching?
If it would be possible to get this information in the next few hours, that'd be great, becasue we're patching and restarting the cluster tonight, so if we're going to install watcher on the main cluster rather than our marvel cluster, we want to do it along with this patching/restart.
Thanks much in advance!