Thanks for the response @stephenb .
I've annotated the pods according to the documentation for pods with multiple containers as we have a number of sidecars alongside our software services and for the time being we're not interested in their logs:
Multiple containersedit
When a pod has multiple containers, the settings are shared unless you put the container name in the hint. For example, these hints configure multiline settings for all containers in the pod, but set a specific exclude_lines hint for the container called sidecar.
annotations:
co.elastic.logs/multiline.pattern: '^\['
co.elastic.logs/multiline.negate: true
co.elastic.logs/multiline.match: after
co.elastic.logs.sidecar/exclude_lines: '^DBG'
I fear I may have wasted your time a bit here, I did what you suggested and took a look at the reference file. Comparing it to what I posted earlier I ended up removing the "type" and "paths", replacing that block with
hints.default_config.enabled: false
After doing this I noticed errors in the filebeat logs mentioning the Redis Module inputs so I removed that entire block as well, restarted Filebeat and the autodiscover is now working 
My final Filebeat config is now simply:
filebeat.autodiscover:
providers:
- type: kubernetes
hints.enabled: true
hints.default_config.enabled: false
Many thanks for the help and for pointing me in the right direction 