Mutate filter plugin: convert to keyword and text?

Elastic Stack Logstash
1

From the current mutate convert documentation:

Valid conversion targets are: integer, float, string, and boolean.

I expected to see the new-for-5.0 keyword and text types in that list. And perhaps a note about the string type being deprecated.

I was about to create an issue for the logstash-filter-mutate GitHub repo about this, but after reading the following request in the placeholder text for the new issue:

Please post all product and debugging questions on our forum

I decided to create this forum topic.

2

You're conflating the types of fields in Logstash events with the mappings in Elasticsearch. They are entirely different things. You should think of Logstash events as JSON documents, and JSON values are either objects, arrays, strings, numbers, or booleans. How these values are mapped in Elasticsearch is decided by Elasticsearch and is nothing Logstash can affect except via the index template (and indirectly by converting fields to a JSON data type that allows ES's dynamic mapper to do the right thing).

1 Like
3

D'oh! Thank you. Yes, I see what you mean. And, yes, you're right about what I was thinking. I'm embarrassed, but grateful for the correction.

Using mutate to convert the data type of a field to the Elasticsearch-specific keyword or text type was an idea (a wrongheaded one, as you've pointed out) that occurred to me as a possible alternative to what I'm actually doing, which is to use an index template.

What I was doing in Elastic < 5.0

In Elastic 1.x and 2.x, I used the following index template to set all string fields in the index pattern fuw-* to not_analyzed:

{
  "template" : "fuw-*",  1 
  "mappings" : {
    "_default_" : {
    "dynamic_templates" : [ {
      "string_fields" : {
      "mapping" : {
        "index" : "not_analyzed",
        "omit_norms" : true,
        "type" : "string"
      },
      "match_mapping_type" : "string",
      "match" : "*"
      }
    } ]
    }
  }
}

(This works for me, and I'll likely still need to use/support Elasticsearch 2.x, so I'd gratefully accept tips.)

What I'm doing now in Elastic 5.0

In Elastic 5.0, I understand that the index property no longer supports the value not_analyzed.

So I'm now using the following index template to map "incoming" strings to the new-for-5.0 keyword type:

{
  "template": "fuw-*",
  "mappings": {
    "_default_": {
      "dynamic_templates": [{
        "string_fields": {
          "match": "*",
          "match_mapping_type": "string",
          "mapping": {
            "type": "keyword"
          }
        }
      }]
    }
  }
}

This seems to be working: for example, after loading data into that index pattern, Kibana doesn't show separate "keyword" fields in that index pattern, and I can successfully search on the "original" (non-analyzed) values.

But I'm left with a lingering doubt: in Management / Indices / Index Patterns, Kibana still shows the types of these fields in that index pattern as string.

I expected to see these fields as type keyword.

From the description at the top of that Kibana page:

This page lists every field in the fuw-* index and the field's associated core type as recorded by Elasticsearch.

I Googled for "core type", and found the Elastic documentation topic "Field datatypes":

Core datatypes

string
text and keyword

Based on that documentation, perhaps everything's working as designed, and Kibana is correctly reporting string as the core type, where text and keyword are types that "belong" to that core type?

Or am I, once again, conflating types from different contexts?

4

My apology for my poor forum etiquette: it belatedly occurred to me that I should have closed this topic, and created a new topic for my "follow-on" question about Kibana reporting the "core type" as string. I'll do that - create a new topic - if asked, no problem.

5

I'm not 100% sure if keyword fields are supposed to be reported as strings. Asking in the Kibana or Elasticsearch group is probably better.

6

Done (thanks for the tip):

Kibana 5.0 shows field type as string, not keyword: working as designed?

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.