Unexpected fielddata cache usage

I have several indexes pointing to same alias. Each index has different type of documents stored in it. I do a query that sorts documents of some type by _uid. When doing this query I noticed that it takes a lot of memory in field data cache. I expect this query to only fill cache with data of indexes with matching type. Instead it seems it fills cache with data from all indexes. Could someone explain why is it happening?