Fielddata getting set to 0 bytes in a few seconds

I'm not sure if this is normal in ES2, but in ES1 this is not happening.

I don't have anything in my config, with field data, ie indices.fielddata.cache.size. indices.breaker.fielddata.limit, etc.

When I perfrom a search, I see some fielddata activity in _nodes/stats/indices/fielddata?fields=* and _cat/fielddata but after a few seconds they are down to 0. The evictions is constantly 0.

breakers tripped: 0
evictions: 0

I can see in _nodes/stats/indices/fielddata?fields=*:

fielddata: {
memory_size_in_bytes: 11736472,

then refresh, and it's:
memory_size_in_bytes: 3912,

Also, here's what the marvel chart looks like:

What could be causing this?