# ILM, Fielddata memory error on Warm nodes

**URL:** <https://discuss.elastic.co/t/ilm-fielddata-memory-error-on-warm-nodes/303916>\
**Category:** Elasticsearch\
**Created:** [May 4, 2022, 8:44am UTC](https://discuss.elastic.co/t/ilm-fielddata-memory-error-on-warm-nodes/303916 "2022-05-04T08:44:22Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![benoitp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benoitp/32/97430_2.png) [@benoitp](https://discuss.elastic.co/u/benoitp)\
**Post date:** [May 4, 2022, 8:44am UTC](https://discuss.elastic.co/t/ilm-fielddata-memory-error-on-warm-nodes/303916/1 "2022-05-04T08:44:22Z")

</div>

Hello

I'm trying to understand something in the logic of Elasticsearch.

Here is my setup :  
Graylog 4.2 cluster for ingesting logs  
Connected to an ES 7.10 cluster  
The Graylog cluster knows the ES Cluster by the Master nodes who are set in the Graylog's config files (as "discovery.seed\_hosts" for thoses who knows Graylog).

![graylog to ES arch](https://us1.discourse-cdn.com/elastic/original/3X/6/a/6a23787bcdc2edc2ea71e93c3d58f08f9e7d9068.png)

I have an index rotation based on time, one indice per week.  
With ILM I have 4 weeks of logs on the Hot data nodes.  
After 4 weeks the indices are moved to the Warm nodes.  
On the Warm nodes, 22 weeks are keeped, the older are dropped.

In the indice template, the "source" field who contain the hostname who sent the log is set as "fieldata"

```auto
"source": {
  "fielddata": true,
  "analyzer": "analyzer_keyword",
  "type": "text"
},

```

Some example of hostnames I have in the "source" field :

```auto
pirv-siem-relay-01
dirv-monitoring-centreon-02
eidv-vrouter-unitary11-ma-02

```

My question is the following :  
When I run a dashboard in Graylog **on the LAST 5 MINUTES of logs** , which is doing aggregration based on the "source" field I have a fielddata memory error on the Warm nodes :

`[2022-05-03T18:17:17,946][WARN][o.e.i.b.fielddata] [eirv-siem-es-07] [fielddata] New used memory 7165570335 [6.6gb] for data of [source] would be larger than configured breaker: 6871947673 [6.3gb], breaking`

If I enlarge the HEAP size only on the Warm nodes, the error disappears.

I understand the error, but what i don't understand is why the error is happening on the Warm nodes, for a request that are only searching in the last 5 minutes of logs.  
The request is not supposed to run only on the Hot nodes ?

Thanks for reading.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [May 5, 2022, 5:31am UTC](https://discuss.elastic.co/t/ilm-fielddata-memory-error-on-warm-nodes/303916/2 "2022-05-05T05:31:24Z")

</div>

Hi @benoitp

Just a couple thoughts ...

If you have fields that like source that you describe that you want to aggregate on .. they should be of type `keyword` not `text` that would probably solve this problem.

Think of `keyword`s as fields you want to aggregate on or filter on or both... exact matches etc, categories, hostname, account id, host type, os type etc.. etc.. `keywords` are extremely efficient for this types of data and operations

Think of `text` as a free text search like on a description or a message etc.

Using a `text` data types to aggregate on is a not a best practice for Elasticsearch

2nd... Sending all your queries and / or ingest through your master nodes.. if that is really what the picture is showing is also a bit of an anti pattern.. the ingest / queries should be pointing at your hot data nodes ... or better yet you could create ingest / coordinator nodes and point all your ingest / queries to them...  
Going through you masters will work, but all they do is turn around and forward those request to a data node and you are using some valuable resources on the master nodes.

---

<div class="post-metadata">

**Author:** ![benoitp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benoitp/32/97430_2.png) [@benoitp](https://discuss.elastic.co/u/benoitp)\
**Post date:** [May 5, 2022, 8:01am UTC](https://discuss.elastic.co/t/ilm-fielddata-memory-error-on-warm-nodes/303916/3 "2022-05-05T08:01:04Z")

</div>

Hi @stephenb

When using a field type of `keywords` , can I search or do aggregation for a only a part of the string ?  
Like "eidv-vrouter" or "eidv-vrouter\*" to get all the messages from a group of sources ?  
As you are talking about "exact matches" I'm wondering about this.  
I read the doc but I didn't find a precise answer. 😟

About the master nodes, indeed I have a lot of traffic on it and I thought it was the normal behavior. I will dig the idea of the ingest / coordinator nodes.

Thank you for sharing your thoughts.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [May 5, 2022, 2:30pm UTC](https://discuss.elastic.co/t/ilm-fielddata-memory-error-on-warm-nodes/303916/4 "2022-05-05T14:30:06Z")

</div>

> [@benoitp](#):
>
> Like "eidv-vrouter" or "eidv-vrouter\*" to get all the messages from a group of sources ?  
> As you are talking about "exact matches" I'm wondering about this.  
> I read the doc but I didn't find a precise answer. 😟

Yes you can... There are a couple ways here is one, but it also depends on overall what you are trying to do.. you _can_ you field data it is just going to require a lot of memory and not be that efficient

Here is one way this searches for sources that begin with `eidv` and then aggregates the results

> **[Prefix query | Elasticsearch Guide \[8.11\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-prefix-query.html)**

```auto
GET filebeat-*/_search
{
  "size": 0, 
  "query": {
    "prefix": {
      "source": {
        "value": "eidv"
      }
    }
  },
  "aggs": {
    "apps": {
      "terms": {
        "field": "source",
        "size": 10
      }
    }
  }
}

```

You could also use wildcard

> **[Wildcard query | Elasticsearch Guide \[8.11\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-wildcard-query.html)**

```auto
GET filebeat-*/_search
{
  "size": 0, 
  "query": {
    "wildcard": {
      "source": {
        "value": "eidv*"
      }
    }
  },
  "aggs": {
    "apps": {
      "terms": {
        "field": "source",
        "size": 10
      }
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 2, 2022, 2:30pm UTC](https://discuss.elastic.co/t/ilm-fielddata-memory-error-on-warm-nodes/303916/5 "2022-06-02T14:30:22Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.