Rename Field in Index (index, visu, dashboard, etc...)

(Guillaume Baudringhien) #1

Hi all,
Thanks in advance for the help!
I want to rename a field in an index 'everywhere'. By everywhere, I mean that I don't want my graphs and dashboard based on this field to be impacted, I don't know if there is any possibility to do it.
I found this like on the web : https://stackoverflow.com/questions/43120430/elasticsearch-mapping-rename-existing-field
It's using an Ingest pipeline, a Rename processor and the Reindex API but I don't want to create a new index because I have more than hundred graphs included in many dashboards (means : very long to update everything with the new name of my field)
Thanks again for your help !
Regards
Guillaume

[EDIT] - quick edit just to say that i'm running elasticsearch 6.4.3 and also to say why do I want to rename a field. For the moment I have something like :

  • logs before 10th Dec I have into them myOldFieldName = user1
  • logs after 10th Dec I have into them myNewFieldName = user1
    and I'd like the same myNewFieldName = user1 for all my logs
(David Pilato) #2

Would that help? https://www.elastic.co/guide/en/elasticsearch/reference/6.4/alias.html

(Guillaume Baudringhien) #3

Hello @dadoonet and thank you for your answer,
I will take a look at it as soon as I can but first of all, I didn't see your reply before editing my post.
Did you see in my post the edit below :

Many thanks

(David Pilato) #4

I do see the edit.

(Guillaume Baudringhien) #5

Hi @dadoonet and all,
I've been reading the page that you told me but I'm not sure if it can work in my case ...
My command GET /_search returns :
{
"my-index-name":{
"aliases":{},
"mappings":{
"spread_doctype":{
"properties":{
"@timestamp":{
"type":"date"
},
"@version":{
"type":"text",
"fields":{
"keyword":{
"type":"keyword",
"ignore_above":256
}
}
},
"my-old-field-name":{
"type":"text",
"fields":{
"keyword":{
"type":"keyword",
"ignore_above":256
}
}
},
"my-new-field-name":{
"type":"text",
"fields":{
"keyword":{
"type":"keyword",
"ignore_above":256
}
}
}
}
}
},
"settings":{
"index":{
//some settings
}
}
}
}
As you can see, my-old-field-name and my-new-field-name are two independent fields in this index but the old one for the old logs and the new one for the new logs. I don't want to create a third field as alias here, I just wanna merge these two fields into a unique one.
Sorry If I'm not clear
Thank you

(xeraa) #6
  1. Please format your code for better readability :slight_smile:
  2. While you are saying that this is the output of GET /_search it looks like a mapping to me?
  3. You said earlier that "logs before 10th Dec I have into them myOldFieldName = user1". Do you have a daily / weekly / ... index pattern? Then you would just add the alias to indices before the 10th of December. If it's one big index you'll probably need to do an _update_by_query on all fields that don't have the field myNewFieldName. Shout if the query with the script to rename the field is not obvious. At the end of https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html there is a very similar script, but you won't need a full reindex — update by query on the old format will do it.
(Guillaume Baudringhien) #7

Hi @xeraa and thanks for your time,

  1. Sorry I really don't figure out how the formatted text works ....

  2. Below the answer for GET /spread/_mapping/ (you were right)
    {
    "my-index-name":{
    "aliases":{},
    "mappings":{
    "spread_doctype":{
    "properties":{
    "@timestamp":{
    "type":"date"
    },
    "@version":{
    "type":"text",
    "fields":{
    "keyword":{
    "type":"keyword",
    "ignore_above":256
    }
    }
    },
    "my-old-field-name":{
    "type":"text",
    "fields":{
    "keyword":{
    "type":"keyword",
    "ignore_above":256
    }
    }
    },
    "my-new-field-name":{
    "type":"text",
    "fields":{
    "keyword":{
    "type":"keyword",
    "ignore_above":256
    }
    }
    }
    }
    }
    },
    "settings":{
    "index":{
    //some settings
    }
    }
    }
    }

  3. My logs are imported by logstash to elasticearch every minute from a csv file. Before this 10th Dec the header of this csv file was with myOldFieldName field and after this 10th Dec I modified this header with myNewFieldName. So now what happened ? My logs before and after this 10th Dec don't have the same field name for the same "attribute". It's very painful in the case I want to create a dashboard and apply a filter based on this field ...

So your solution would be to add an alias on my logs that don't already contain myNewFieldName ? What do you mean by "big index" ? Mine has almost 30M logs ... I won't get any issue by adding an alias with the same name than a real field name contained in other (newer) logs ?

Many thanks

(xeraa) #8

So the way I read this is that you have a single index spread with all the data, right? Then you will need the _update_by_query approach for the old data. Otherwise you'd have the mapping problem of colliding a concrete field and the alias field.

(Guillaume Baudringhien) #9

Yes I'm only interested on this index. It doesn't have any link with the others.
Yeah totally

(Guillaume Baudringhien) #10

Hi,
I hope you re doing well,
I've found how to create an alias but not how to say that I want to create this alias for the logs before a specific date or where a field doesn't exist or exist ...

I mean, I would like to do :
create my alias
PUT myIndex
{
"mappings": {
"_doc": {
"properties": {
"myOldFieldName": {
"type": "keyword"
},
"myNewFieldName": {
"type": "alias",
"path": "myOldFieldName"
}
}
}
}
}
only for those logs
GET /myIndex/_search
{
"query": {
"range" : {
"@timestamp": {
"lte" : "dateOfTheUpdate"
}
}
}
}

(xeraa) #11
  1. Please format your code.
  2. You are using an index alias. I think what you actually would need is a field alias if the new field doesn't exist anywhere yet:
DELETE field_alias

PUT field_alias/_doc/1
{
  "foo": "test"
}

GET field_alias/_mapping

GET field_alias/_search
{
  "query": {
    "match": {
      "bar": "test"
    }
  }
}

PUT field_alias/_mapping/_doc
{
  "properties": {
    "bar": {
      "type": "alias",
      "path": "foo"
    }
  }
}

GET field_alias/_mapping

GET field_alias/_search
{
  "query": {
    "match": {
      "bar": "test"
    }
  }
}
  1. If the field already exists in the index you will need to run an update query (though this is relatively heavy depending on how many documents need to be reindexed):
DELETE update_index

PUT update_index/_doc/1
{
  "foo": "test"
}
PUT update_index/_doc/2
{
  "bar": "test"
}

GET update_index/_mapping

GET update_index/_search
{
  "query": {
    "match": {
      "foo": "test"
    }
  }
}

# This will fail with: mapper [bar] of different type, current_type [text], merged_type [FieldAliasMapper]
PUT update_index/_mapping/_doc
{
  "properties": {
    "bar": {
      "type": "alias",
      "path": "foo"
    }
  }
}

POST update_index/_update_by_query
{
  "query": {
    "bool": {
      "must": {
        "exists": {
          "field": "bar"
        }
      },
      "must_not": {
        "exists": {
          "field": "foo"
        }
      }
    }
  },
  "script": {
    "source": """
      ctx._source.foo = ctx._source.bar;
    """
  }
}

GET update_index/_search
{
  "query": {
    "match": {
      "foo": "test"
    }
  }
}
  1. Depending on your scenario you could also solve the problem at query time by simply searching both fields:
DELETE query_time

PUT query_time/_doc/1
{
  "foo": "test"
}
PUT query_time/_doc/2
{
  "bar": "test"
}

GET query_time/_search
{
  "query": {
    "multi_match": {
      "query": "test",
      "fields": [ "foo", "bar"]
    }
  }
}