Proper Template Mapping for ES/Kibana 6.2.3

Elastic Stack Kibana
1

We are migrating from ES 2.4/Kibana4 to ES/Kibana 6.2.3. We use our own tool for pushing structured JSON into the ElasticSearch cluster. Functionally everything is basically working .. but we're struggling with the right mappings in the latest version of ES.

Here is the index template we currently have:

{
  "order": 99,
  "index_patterns": [
    "logs-*"
  ],
  "settings": {
    "index": {
      "codec": "best_compression",
      "refresh_interval": "30s",
      "number_of_shards": "52",
      "number_of_replicas": "1"
    }
  },
  "mappings": {
    "event": {
      "_source": {
        "enabled": true
      },
      "properties": {
        "all": {
          "type": "keyword",
          "index": true
        }
      },
      "dynamic_templates": [
        {
          "geo_fields": {
            "path_match": "*__geo.location",
            "mapping": {
              "type": "geo_point"
            }
          }
        },
        {
          "epoch_fields": {
            "path_match": "*epochms",
            "mapping": {
              "type": "date",
              "doc_values": true
            }
          }
        },
        {
          "all_strings": {
            "match_mapping_type": "string",
            "mapping": {
              "copy_to": "all",
              "type": "text",
              "index": true,
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "index": true,
                  "ignore_above": 256
                }
              }
            }
          }
        }
      ]
    }
  },
  "aliases": {}
}

We have a few goals with this template:

  • All string fields should be searchable (obviously).
  • Any string field thats shorter than 256chars should be usable in visualizations. We may shorten this in the future.
  • Any string field longer than 256chars is not
  • All tokenized words are sent into an all field, that we pass to Kibana as the default_field when people do string searches that have no field name associated with them.

I think we've got this maybe half right .. but not quite right. We have a few notable problems right now that we could use help with.

all field searching is only partially working

In ES 2.4/Kibana2.4, if we search for a partial string (say "cron:session"), we get back a highlighted match anywhere in our documents with that string.. whether its a full match to a field or a partial match.

With our current mapping in ES 6, we actually get no results on that search. If we search for a value that maps to an entire field (say cron) we will get a match back because some documents have program__str: "cron" in them. Also, this search is case-sensitive .. which is not ideal.

I will note, if we search for message__str:"cron:session", the results are good.. this failure only applies to the all field.

highlighting does not work

When we search for a match in a specific field, highlighting works.. but when we use the all field (above), if we happen to get a match back, we will still not get highlighting of that match. Seems like this may be likely related to the above issue?

2

At a first glance, what I notice is that the all field has been mapped as type keyword. As a result, this field only supports exact (case sensitive) matches. I think the issues you're seeing can be solved by changing the mapping of that field to type text.

Note that you can't actually change the mapping of a field on an existing index. You'll need to create a new index with the updated mapping and reindex your data to that new index.

1 Like
3

Indeed - your right. I figured that out last night and tweaked the mapping and we are able to see that the biggest problem of the field-less text searches is fixed. The second issue of highlighting isn't fixed.

4

Still looking for help with the highlighting -- or at least understanding whether or not that should work in the scenario of searching without a field name?

5

I think overriding default_field may interfere with highlighting. I'd advise you create a new topic so that your question will have better visibility amongst the Kibana developers that may have some insights.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.