Hello,
Our product Quepid allows you tune relevance while immediately measuring the impact of your changes on 1000s of queries. Basically how it works is you point Quepid at your Elasticsearch instance. After adding queries, you write tests or make judgements on your ideal search results for those queries. As you modify your Query DSL to tune, you can see the impact immediately on your searches.
We believe strongly in performing test-driven relevance to measure the impact of any changes. We've been doing this with Solr for years, in Quepid, and we're happy to bring this product to the Elasticsearch crowd.
More info in this blog post. Feel free to ping me with any questions, complaints, or the like.