# Keyword fields, Graylog and ElastAlert

**URL:** <https://discuss.elastic.co/t/keyword-fields-graylog-and-elastalert/165877>\
**Category:** Elasticsearch\
**Created:** [January 27, 2019, 4:10am UTC](https://discuss.elastic.co/t/keyword-fields-graylog-and-elastalert/165877 "2019-01-27T04:10:53Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![shortstack](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shortstack/32/100341_2.png) [@shortstack](https://discuss.elastic.co/u/shortstack)\
**Post date:** [January 27, 2019, 4:10am UTC](https://discuss.elastic.co/t/keyword-fields-graylog-and-elastalert/165877/1 "2019-01-27T04:10:53Z")

</div>

have graylog in front of elasticsearch, getting the following error when trying to generate quick values:

```auto
Unable to perform search query\n\nFielddata is disabled on text fields by default. Set fielddata=true on [param_name] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.

```

it sounds like the latter is the best route, but wondering what the implications will be. this only involves changing the index mapping, as far as i'm understanding? also, trying to wrap my head around the .keyword vs .raw concept. because we also use elastalert. would like to know how this will affect those rules, and performance, or anything else glaring we should be aware of.

---

<div class="post-metadata">

**Author:** ![forloop](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/forloop/32/9021_2.png) [@forloop](https://discuss.elastic.co/u/forloop)\
**Post date:** [January 27, 2019, 4:51am UTC](https://discuss.elastic.co/t/keyword-fields-graylog-and-elastalert/165877/2 "2019-01-27T04:51:12Z")

</div>

Certain operations in Elasticsearch such as sorting, aggregations and access to field values in scripts, require looking up the terms associated with a document field. This is a different data access pattern than what is needed for full-text search, which needs to search a collection of terms and find the associated documents.

With a `text` field datatype mapping, index time [analysis](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis.html) controlled by the analyzer associated with the field mapping (or the default [`standard` analyzer](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-standard-analyzer.html), if none is defined), will tokenize the input and produce _0-to-many_ terms that will be inserted into the inverted index. The inverted index is _the_ data structure for full-text search. When it comes to one of the aforementioned operations like sorting and aggregations, the terms in an inverted index would need to be _un-inverted_ in memory to build a columnar data structure, that can then be searched efficiently to provide values needed for the operation. Elasticsearch does not allow this _un\_inversion_ process for `text` datatypes, without explicitly setting `fielddata` to `true` because, for a large set of documents, it is possible that this in memory data structure is bigger than the heap allocated to Elasticsearch, potentially leading to the node process running out of memory.

Now, with a `keyword` field datatype mapping, this columnar data structure can be built and persisted to disk at index time, and is known by its Lucene name, [`doc_values`](https://www.elastic.co/guide/en/elasticsearch/reference/current/doc-values.html) (Check out the [deep dive on doc\_values](https://www.elastic.co/guide/en/elasticsearch/guide/current/_deep_dive_on_doc_values.html) if you're interested to know more). When it comes to an operation that needs to use `doc_values`, these data structures can be loaded on demand, and leverage the filesystem cache for quick access, no longer constrained by the size of the heap allocated to Elasticsearch.

So, for a string in a JSON object, which one do we use, `text` or `keyword`? Well, it depends on what you need to do with the field in Elasticsearch

1. Need to perform full-text search? Use `text` datatype
2. Need to perform term-level matching (like looking for **exact values** ), sorting, aggregations? Use a `keyword` datatype

Thankfully with Elasticsearch, we don't need to pick one or the other, we can index the field as both a `text` datatype **and** a `keyword` datatype, then use the correct field for the respective operations! This is where [`fields`](https://www.elastic.co/guide/en/elasticsearch/reference/current/multi-fields.html) a.k.a. `multi_fields` come in. In the mapping for field, we can specify that the field should be indexed in multiple ways

```json
PUT my_index
{
  "mappings": {
    "_doc": {
      "properties": {
        "param_name": {
          "type": "text",
          "fields": {
            "keyword": { 
              "type": "keyword",
              "ignore_above": 256
            }
          }
        }
      }
    }
  }
}

```

This maps `param_name` as both a `text` datatype and `keyword` datatype; when we need to target a field for full-text search, we can use `"param_name"`, and when we need to target a field for sorting, aggregations, or the operation in question, we can use `"param_name.keyword"`.

The mapping example that I've shown is the [default inferred mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-field-mapping.html) for a string field in a JSON object, so if you have not explicitly mapped the type in the given index, you should have a `"param_name.keyword"` that you can perform the operation against.

> [@shortstack](#):
>
> it sounds like the latter is the best route, but wondering what the implications will be. this only involves changing the index mapping, as far as i'm understanding?

Either option, you will need to change the mapping. Since this is a mapping change that would affect already indexed documents, you would need to reindex the existing documents for the mapping change to affect them. The [Reindex API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html) can help with this.

The _preferred_ option would be to map as a `keyword` datatype, since it can leverage `doc_values`. You may already have this `keyword` mapping though; take a look at the mapping that you have in the index with

```json
GET /<index_name>/_mapping

```

> [@shortstack](#):
>
> also, trying to wrap my head around the .keyword vs .raw concept. because we also use elastalert. would like to know how this will affect those rules, and performance, or anything else glaring we should be aware of.

I'm not familiar with ElastAlert, but essentially, `.keyword` and `.raw` are just different names for a `multi_field` on a mapping; In versions of Elasticsearch before 5.x, `raw` was the name typically used for a `not_analyzed` `string` field (which is what the `keyword` datatype is in versions prior to 5.x).

---

<div class="post-metadata">

**Author:** ![shortstack](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shortstack/32/100341_2.png) [@shortstack](https://discuss.elastic.co/u/shortstack)\
**Post date:** [January 31, 2019, 4:28pm UTC](https://discuss.elastic.co/t/keyword-fields-graylog-and-elastalert/165877/3 "2019-01-31T16:28:35Z")

</div>

Thank you for the detailed response. This is very helpful.

We have a custom mapping, and new indices are being created with that field as a keyword type, instead of text. I added the below:

```
	  "param_name" : {
		"type" : "keyword"
	  }

```

When searching the newly created indices with that mapping applied, I am still receiving the "Fielddata is disabled on text fields by default. " message, even though I can see that field when I query the index mapping in ES and see that it is in fact a keyword now instead of text.

Is there something obvious I'm missing?

---

<div class="post-metadata">

**Author:** ![forloop](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/forloop/32/9021_2.png) [@forloop](https://discuss.elastic.co/u/forloop)\
**Post date:** [February 1, 2019, 5:25am UTC](https://discuss.elastic.co/t/keyword-fields-graylog-and-elastalert/165877/4 "2019-02-01T05:25:11Z")

</div>

Would you be able to show the search you're using, and the mapping(s) for the index/indices you're targeting?

It sounds like there's a `param_name` field somewhere that is still mapped as `text`.

---

<div class="post-metadata">

**Author:** ![shortstack](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shortstack/32/100341_2.png) [@shortstack](https://discuss.elastic.co/u/shortstack)\
**Post date:** [February 1, 2019, 2:58pm UTC](https://discuss.elastic.co/t/keyword-fields-graylog-and-elastalert/165877/5 "2019-02-01T14:58:19Z")

</div>

that's what i thought, too

have done this GET on all new indices created, and see the parameter being mapped as keyword

GET index-name/\_mapping?pretty

however, i perform the search in graylog:

```auto
_exists_:param_name AND _index:index-name

```

i try to show quick values on the param\_name field, i get the 500 complaining about it being a text field

```auto
{
  "index-name": {
    "mappings": {
      "message": {
        "dynamic_templates": [
          {
            "internal_fields": {
              "match": "gl2_*",
              "mapping": {
                "type": "keyword"
              }
            }
          },
          {
            "store_generic": {
              "match_mapping_type": "string",
              "mapping": {
                "type": "keyword"
              }
            }
          }
        ],
        "properties": {
          "BASE10NUM": {
            "type": "keyword"
          },
          "GREEDYDATA": {
            "type": "keyword"
          },
          "HOSTNAME": {
            "type": "keyword"
          },
          "IPORHOST": {
            "type": "keyword"
          },
          "MONTHDAY": {
            "type": "keyword"
          },
          "MONTHNUM": {
            "type": "keyword"
          },
          "NUMBER": {
            "type": "keyword"
          },
          "POSINT": {
            "type": "keyword"
          },

          ....
          
          "param_name": {
            "type": "keyword"
          },

          ....

          
        }
      }
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![forloop](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/forloop/32/9021_2.png) [@forloop](https://discuss.elastic.co/u/forloop)\
**Post date:** [February 2, 2019, 10:21am UTC](https://discuss.elastic.co/t/keyword-fields-graylog-and-elastalert/165877/6 "2019-02-02T10:21:23Z")

</div>

Are there any indices that existed before the mapping change was made, that are being targeted by the query?

The following query will get all the mappings for all indices and use [`filter_path`](https://www.elastic.co/guide/en/elasticsearch/reference/current/common-options.html#common-options-response-filtering) to include only paths to `param_name` field type mappings

```auto
curl -X GET "localhost:9200/_mapping?pretty&filter_path=**.properties.param_name.type"

```

which will return something like

```json
{
  "index-name-1" : {
    "mappings" : {
      "message" : {
        "properties" : {
          "param_name" : {
            "type" : "keyword"
          }
        }
      }
    }
  },
  "index-name-2" : {
    "mappings" : {
      "message" : {
        "properties" : {
          "param_name" : {
            "type" : "text"
          }
        }
      }
    }
  }
}

```

This should make it a bit easier to spot which index has `param_name` mapped as `"text"`.

---

<div class="post-metadata">

**Author:** ![shortstack](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shortstack/32/100341_2.png) [@shortstack](https://discuss.elastic.co/u/shortstack)\
**Post date:** [February 3, 2019, 6:30pm UTC](https://discuss.elastic.co/t/keyword-fields-graylog-and-elastalert/165877/7 "2019-02-03T18:30:02Z")

</div>

there are hundreds that were previously mapped as text, but in my query i'm specifically searching the new index (with the keyword field mapping, not text). so i'm confused as to why it's not seeing that.

---

<div class="post-metadata">

**Author:** ![forloop](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/forloop/32/9021_2.png) [@forloop](https://discuss.elastic.co/u/forloop)\
**Post date:** [February 3, 2019, 10:17pm UTC](https://discuss.elastic.co/t/keyword-fields-graylog-and-elastalert/165877/8 "2019-02-03T22:17:34Z")

</div>

What about if you use the following query

```auto
_exists_:"param_name" AND _index:"index-name"

```

---

<div class="post-metadata">

**Author:** ![shortstack](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shortstack/32/100341_2.png) [@shortstack](https://discuss.elastic.co/u/shortstack)\
**Post date:** [February 4, 2019, 1:52pm UTC](https://discuss.elastic.co/t/keyword-fields-graylog-and-elastalert/165877/9 "2019-02-04T13:52:27Z")

</div>

that's exactly what i'm doing: [Keyword fields, Graylog and ElastAlert](https://discuss.elastic.co/t/keyword-fields-graylog-and-elastalert/165877/5?u=shortstack)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 4, 2019, 2:00pm UTC](https://discuss.elastic.co/t/keyword-fields-graylog-and-elastalert/165877/10 "2019-03-04T14:00:33Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.