Reserved field names

Elastic Stack Elasticsearch
1

We have been planning to upgrade from 1.7.1 to 2.3 but we do not seem to pass the Migration Checker
I am having two issues marked as Red and I do not know how to address them yet.
Please help.
Some of the indices are flagged using Reserved field names:
issue 1:
Reserved field names
The _uid, _id, _type, _source, _all, _parent, _field_names, _routing, _index, _size, _timestamp, and _ttl field names are reserved and can no longer be used in the document _source., in type: logs.

issue 2:
Conflicting field mappings
Mapping for field dashboard:hits conflicts with: search:hits. Check parameter: type
Please help

2

Hi,

  • issue1: the mentioned field names appear somewhere in your documents, but are not legal in 2.x any more, so you need to rename those fields. You need to reindex your data, also because of the following issue.

  • issue2: this means you two different mappings for the field "hits" in type dashboard and in type "search". This is no longer possible with ES 2.x. This blog article explains why. Unfortunately the only solution is to reindex your data into a new index with corrected mappings

3

Hi Chris
I am new to ELK, I just found out that we do not use these reserved field names ( uid, _id, _type, _source, _all, _parent, _fieldnames, _routing, _index, _size, _timestamp), what is the best way to remove them?

4

Please help
I try to use the filter and rename as listed below to re-index using logstash but it does not work

mutate {
rename => { "_uid" => "uid" }
rename => { "_id" => "id" }
rename => { "_type" => "type" }
rename => { "_source" => "source" }
rename => { "_all" => "all" }
rename => { "_parent" => "parent" }
rename => { "_field_names" => "fielnames" }
rename => { "_routing" => "routing" }
rename => { "_index" => "index" }
rename => { "_size" => "size" }
rename => { "_timestamp" => "timestamp" }
rename => { "_ttl" => "ttl" }
}
I ran the migration check and it still showing
Reserved field names
The _uid, _id, _type, _source, _all, _parent, _field_names, _routing, _index, _size, _timestamp, and _ttl field names are reserved and can no longer be used in the document _source., in type: logs.

5

Any help please, we're still dealing with this issue
Thanks

6

I can't really tell what you did, but when you did the transformation you'll need to make a new index and delete the old one. Once you add a field to an index it can never be removed from the mapping.

7

This is what I did to re-index using logstash but it does not work
input {
elasticsearch {
hosts => [ "elasticsearch.prod.xxx:9200" ]
index => "index-2016.05.22"
size => 500
scroll => "5m"
docinfo => true
}
}

filter {
ruby {
init => "
def remove_dots hash
new = Hash.new
hash.each { |k,v|
if v.is_a? Hash
v = remove_dots(v)
end
new[ k.gsub('.','') ] = v
if v.is_a? Array
v.each { |elem|
if elem.is_a? Hash
elem = remove_dots(elem)
end
new[ k.gsub('.','') ] = elem
} unless v.nil?
end
} unless hash.nil?
return new
end
"
code => "
event.instance_variable_set(:@data,remove_dots(event.to_hash))
"
}
mutate {
rename => { "_uid" => "uid" }
rename => { "_id" => "id" }
rename => { "_type" => "type" }
rename => { "_source" => "source" }
rename => { "_all" => "all" }
rename => { "_parent" => "parent" }
rename => { "_field_names" => "fielnames" }
rename => { "_routing" => "routing" }
rename => { "_index" => "index" }
rename => { "_size" => "size" }
rename => { "_timestamp" => "timestamp" }
rename => { "_ttl" => "ttl" }
}
}

output {
elasticsearch {
hosts => [ "remote.cluster" ]
index => "%{[@metadata][_index]}"
document_type => "%{[@metadata][_type]}"
document_id => "%{[@metadata][_id]}"
}
stdout {
codec => "dots"
}

}

8

Nik and team anymore thoughts on this issue?
Thanks