ELK 5.4 - LS reverts to automatic mapping when 'type' field is removed or renamed

Elastic Stack Logstash
1

Hi there, this post spawns from this one : [ELK 5.4 - LS reverts to automatic mapping when 'type' field is removed or renamed - ultimate :-)). I felt the title was not correct and I didn't explain the issue clearly enough.

This is my index template, just simply mapping three fields :

{
  "order": 0,
  "template": "incoming-aa*",
  "settings": {
    "index": {
      "number_of_shards": "10",
      "number_of_replicas": "1",
      "refresh_interval": "5s"
    }
  },
  "mappings": {
    "incoming-aa": {
      "properties": {
        "source_in": {
          "type": "text"
        },
        "source_location": {
          "type": "geo_point"
        },
        "destination_out": {
          "type": "text"
        }
      }
    }
  },
  "aliases": {
    "Incoming-ALL": {}
  }
}

This is my logstash conf file, where I put the type value in a @metadata field to be used at the output level, after which I split the csv line and remove unwanted fields, among which is the type field :

input {
        file {
                path => "/home/*.csv"
                type => "incoming-aa"
                start_position => "beginning"
        }
}

filter {
        if [type] == "incoming-aa" {
		    # write to @metadata so I can use it in the output part
                mutate { add_field => ["[@metadata][type]", "%{type}"] }

                # split the lines
		    csv {
                        separator => ","
                        columns => ["source_in","source_location","destination_out"]
                }

                # remove unnecessary fields
                mutate {
                        remove_field => ["type","@version", "host", "path", "message"]
                }
				
output {
        if [@metadata][type] == "incoming-aa" {
            elasticsearch {
                hosts => ["10.100.200.100"]
                action => "index"
                index => "incoming-aa-%{+YYYY_MM}"
            	}
	}
}

After ingestion, I check the mapping for that index and below is the result. The part where the lines start and end with "**" (I have put that there so you can see it better, in reality the asterisks are not there) was added by LS/ES, so the original mapping I set up in the index template was not used. An automatic mapping was done and it named it "logs".
This only happens when I remove or rename the "type" field ! When I do not remove it, my mapping is used, but then the "type" field appears in Kibana and I don't want that.

{
  "incoming-aa-2017_06": {
    "mappings": {
      "incoming-aa": {
        "properties": {
          "source_in": {
            "type": "text"
          },
          "source_location": {
            "type": "geo_point"
          },
          "destination_out": {
            "type": "text"
          }
        }
      },
**"logs": {**
**            "properties": {**
**              "@timestamp": {**
**                "type": "date"**
**              },**
**              "source_in": {**
**                "type": "text",**
**                "fields": {**
**                  "keyword": {**
**                    "type": "keyword",**
**                    "ignore_above": 256**
**                  }**
**                }**
**              },**
**              "source_location": {**
**                "type": "geo_point"**
**              },**
**              "destination_out": {**
**                "type": "text",**
**                "fields": {**
**                  "keyword": {**
**                    "type": "keyword",**
**                    "ignore_above": 256**
**                  }**
**                }**
**              }**
**            }**
**          }**
**        }**
**      }**
}

What I am looking for is to get red of that type field or to be able to rename it, without LS choosing automatic mapping.

Any ideas are welcome.

2

I'll close this one as there's a bit of history in the other one, please just edit the subject as mentioned :slight_smile: