Thanks for the details. For the cluster with brokers > 1, it sounds to be exactly this ticket. In the original discuss topic we've been looking at the issue and found the problem to be the distribution of state between brokers and metricbeat not asking remote brokers for meta data if it really needs to do so. It's quite a long discussion, but still an interesting read. To my experience with the API, you're quite lucky it did work for you with 0.10.1, as normally it's more by chance if the number of brokers is > 1.
This brings me to the single broker cluster. Which version are you using with the single broker cluster. Did you test with single broker cluster, version 0.10.2 and still found it failing?
Please note, the feature is in beta right now and we are aware of some issues with it. Unfortunately we haven't had time to work on the metricset yet.
Thanks for reporting your experience and versions being used. This discussion adds quite some valuable input for things to look out for potential problems and tests when I finally find time to improve the metricset.