Let's say that the production cluster crushed and not responding to searches or http queries initiated by watcher, will this fact trigger an alert?
If you install Watcher in a separate cluster (can just be a single node cluster), you can detect via a Watch if your production cluster crashed.
If you are using Marvel in production we recommend that you send all that data from your nodes to a separate cluster (can be a small one).
It'd definitely then make sense to install Watcher on that same cluster and add these sorts of Watches to that instance.
In addition to what mvg and warkolm mentioned, we do expect quite a few people to use Watcher to monitor Elasticsearch health and proactively notify the administrator if issues occur.
Today, Marvel is a great way to monitor Elasticsearch over time - it tracks many cluster health and performance metrics over time. We are planning a tighter integration between Marvel and Watcher over time, but for now, we have started providing recipes for creating watches based on Marvel data:
https://www.elastic.co/guide/en/watcher/current/watching-marvel-data.html