No way to do this currently, but this type of functionality is on our radar.
This is probably the best issue to follow for it: https://github.com/elastic/kibana/issues/5647 - so not only can you turn off inheriting filters, but be able to set your own filter per panel.