How to migrate indices from old .raw template to new .keyword mapping

Elastic Stack Logstash
1

I think I ran into a very big problem when I upgraded from ELK 5 to ELK 6. When I look at the Logstash template in ELK 5 I see a keyword mapping with .raw. After the upgrade to ELK 6 the new template has a keyword mapping with .keyword.

Two problems:

  1. All my Kibana visualizations rely on .raw. Is there an easy way to update that?
  2. The much bigger problem is that I have tons of old indices with a .raw mapping. All new indices have .keyword. How can I solves this problem?

This is my old mapping:

"string_fields": {
					   
										 
          "mapping": {
            "fielddata": {
              "format": "disabled"
            },
            "index": "analyzed",
            "omit_norms": true,
            "type": "string",
            "fields": {
              "raw": {
                "ignore_above": 256,
                "index": "not_analyzed",
                "type": "string"
              }
            }
          },
          "match_mapping_type": "string",
          "match": "*"
        }

This is the new mapping:

"string_fields": {
          "match": "*",
          "match_mapping_type": "string",
          "mapping": {
            "type": "text",			
            "norms": false,			 
            "fields": {
              "keyword": {						
                "type": "keyword"
              }
            }
          }			  
        }
2

I don't know if there's a better solution.

In our case, we used the reindex API (https://www.elastic.co/guide/en/elasticsearch) to recreate the old indexes with .keyword.

In Kibana we exported all saved objects to json, replaced .raw with .keyword and imported the modified json file.

Since we use alias on all indexes, the rest was pretty straight forward.

Hope this helps.

3

If you want to stick to .raw you can start managing the index template yourself. Just download the existing template, rename the field, upload the modified template, and disable the elasticsearch output's manage_template option.

1 Like
4

If I decide to reindex what do I need to do? It looks like I have to reindex each index on its own. How can I migrate .raw to .keyword with the reindex API?

5

It looks like I have to reindex each index on its own.

Yeah, probably, but that's easy to script.

How can I migrate .raw to .keyword with the reindex API?

That'll take care of itself. Try reindexing one of the indexes (it won't delete the old index) and you can inspect the results.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.