# Fielddata size is always different in one of the nodes

**URL:** <https://discuss.elastic.co/t/fielddata-size-is-always-different-in-one-of-the-nodes/188009>\
**Category:** Elasticsearch\
**Created:** [June 28, 2019, 9:42am UTC](https://discuss.elastic.co/t/fielddata-size-is-always-different-in-one-of-the-nodes/188009 "2019-06-28T09:42:35Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![vurus](https://avatars.discourse-cdn.com/v4/letter/v/e5b9ba/32.png) [@vurus](https://discuss.elastic.co/u/vurus)\
**Post date:** [June 28, 2019, 9:42am UTC](https://discuss.elastic.co/t/fielddata-size-is-always-different-in-one-of-the-nodes/188009/1 "2019-06-28T09:42:35Z")

</div>

Hi,

We have a cluster of 6 nodes on 3 servers, all mdi, using ~32 gb heap memory with G1GC, with version 6.4.2 and the fielddata limits are the same (60%) on all nodes.

We have been consistently and periodically confronting out of memory issues due to increasing heap usage. While addressing the cause of this problem, we realized that the fielddata size in one specific node, which is not the master, is always as 1.5x-2x as that of all other nodes, as below.

```
curl -XGET 'localhost:9200/_cat/fielddata?v&h=host,node,field,size&fields=text'
host node field size
host1 node1_1 text 3.6gb
host1 node1_2 text 3.6gb
host2 node2_1 text 3.6gb
host2 node2_2 text 5.4gb
host3 node3_1 text 3.6gb
host3 node3_2 text 3.6gb

```

And, here the limits output are the same in node2\_2 and any other node.

Node2\_2

```
curl -XGET 'localhost:9200/_nodes/node2_2/stats?pretty' 2>/dev/null | grep fielddata -A2
        "fielddata" : {
          "memory_size_in_bytes" : 5899379216,
          "evictions" : 0
--
        "fielddata" : {
          "limit_size_in_bytes" : 20615843020,
          "limit_size" : "19.1gb",

```

Node2\_1

```
curl -XGET 'localhost:9200/_nodes/node2_1/stats?pretty' 2>/dev/null | grep fielddata -A2
        "fielddata" : {
          "memory_size_in_bytes" : 3953272232,
          "evictions" : 0
--
        "fielddata" : {
          "limit_size_in_bytes" : 20615843020,
          "limit_size" : "19.1gb",

```

Also, there is no G1GC marking cycle logs in any nodes for a long time except Node2\_2. There is a new log printed almost per 24 hours.

```
[2019-06-28T09:02:30,676][INFO][o.e.m.j.JvmGcMonitorService] [Node2_2] [gc][311694] overhead, spent [328ms] collecting in the last [1s]
[2019-06-28T09:02:41,831][INFO][o.e.m.j.JvmGcMonitorService] [Node2_2] [gc][old][311696][3] duration [9.9s], collections [1]/[10.1s], total [9.9s]/[30.3s], memory [28.6gb]->[21.9gb]/[32gb], all_pools {[young] [384mb]->[48mb]/[0b]}{[survivor] [0b]->[0b]/[0b]}{[old] [28.2gb]->[21.9gb]/[32gb]}
[2019-06-28T09:02:41,831][WARN][o.e.m.j.JvmGcMonitorService] [Node2_2] [gc][311696] overhead, spent [10s] collecting in the last [10.1s]

```

What may possibly cause this difference?

And the second question is that;

To prevent the long and frequent garbage collections, OOMs, or at least decrease the frequency of it, we daily clear all query, request and fielddata caches with the below request.

```
curl -X POST "localhost:9200/*/_cache/clear"

```

Is it normal to do so, or is what we do against the natural course of events, logic of caching, and should that totally be avoided?

I appreciate any help.  
Thanks.

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [July 1, 2019, 9:40am UTC](https://discuss.elastic.co/t/fielddata-size-is-always-different-in-one-of-the-nodes/188009/2 "2019-07-01T09:40:39Z")

</div>

Hey,

is it possible, that this node holds more data than the others. I.e. if one shard requires 1.8 GB of fielddata, than this node may be holding three shards and instead of two shards like all the other nodes?

--alex

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 29, 2019, 9:40am UTC](https://discuss.elastic.co/t/fielddata-size-is-always-different-in-one-of-the-nodes/188009/3 "2019-07-29T09:40:42Z")

</div>

Closed on July 29, 2019, 9:40am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.