Cardinality and Field Data

We're on the 1.4 series of ES and using cardinality aggregations for a few things. Recently we switched to using a hashed field for better performance. As part of that, we noticed that the hashed field is being stored in field data. I was under the impression the hyper log log implementation was specifically brought in to avoid this? The docs state that the cardinality agg uses a fixed memory set, but we're seeing field data rise with the cardinality of the field. Am I misunderstanding how that agg should work? We do no other aggs on that field, and we have confirmed that it's the cardinality agg specifically that causes it.

Cardinality aggregation does not need extra memory to count distinct values. But field data is still needed to read values.