It works on fielddata, which is either what is stored in doc values if you enabled them, or in the index otherwise.
I suspect your NumberFormatException is because you have an field in your index which is mapped as a string on one type and as an integer on another type. This is unfortunately something that Elasticsearch doesn't support and that we will enforce as of 2.0. The only option would be to reindex either each type in a different index or to assign field names in such a way that they will have the same mapping on all types.