Hi there, this post spawns from this one : [ELK 5.4 - LS reverts to automatic mapping when 'type' field is removed or renamed - ultimate :-)). I felt the title was not correct and I didn't explain the issue clearly enough.
This is my index template, just simply mapping three fields :
{
"order": 0,
"template": "incoming-aa*",
"settings": {
"index": {
"number_of_shards": "10",
"number_of_replicas": "1",
"refresh_interval": "5s"
}
},
"mappings": {
"incoming-aa": {
"properties": {
"source_in": {
"type": "text"
},
"source_location": {
"type": "geo_point"
},
"destination_out": {
"type": "text"
}
}
}
},
"aliases": {
"Incoming-ALL": {}
}
}
This is my logstash conf file, where I put the type value in a @metadata field to be used at the output level, after which I split the csv line and remove unwanted fields, among which is the type field :
input {
file {
path => "/home/*.csv"
type => "incoming-aa"
start_position => "beginning"
}
}
filter {
if [type] == "incoming-aa" {
# write to @metadata so I can use it in the output part
mutate { add_field => ["[@metadata][type]", "%{type}"] }
# split the lines
csv {
separator => ","
columns => ["source_in","source_location","destination_out"]
}
# remove unnecessary fields
mutate {
remove_field => ["type","@version", "host", "path", "message"]
}
output {
if [@metadata][type] == "incoming-aa" {
elasticsearch {
hosts => ["10.100.200.100"]
action => "index"
index => "incoming-aa-%{+YYYY_MM}"
}
}
}
After ingestion, I check the mapping for that index and below is the result. The part where the lines start and end with "**" (I have put that there so you can see it better, in reality the asterisks are not there) was added by LS/ES, so the original mapping I set up in the index template was not used. An automatic mapping was done and it named it "logs".
This only happens when I remove or rename the "type" field ! When I do not remove it, my mapping is used, but then the "type" field appears in Kibana and I don't want that.
{
"incoming-aa-2017_06": {
"mappings": {
"incoming-aa": {
"properties": {
"source_in": {
"type": "text"
},
"source_location": {
"type": "geo_point"
},
"destination_out": {
"type": "text"
}
}
},
**"logs": {**
** "properties": {**
** "@timestamp": {**
** "type": "date"**
** },**
** "source_in": {**
** "type": "text",**
** "fields": {**
** "keyword": {**
** "type": "keyword",**
** "ignore_above": 256**
** }**
** }**
** },**
** "source_location": {**
** "type": "geo_point"**
** },**
** "destination_out": {**
** "type": "text",**
** "fields": {**
** "keyword": {**
** "type": "keyword",**
** "ignore_above": 256**
** }**
** }**
** }**
** }**
** }**
** }**
** }**
}
What I am looking for is to get red of that type field or to be able to rename it, without LS choosing automatic mapping.
Any ideas are welcome.
