ELK 5.4 - LS reverts to automatic mapping when 'type' field is removed or renamed - ultimate :-)

Elastic Stack Logstash
1

Hi,

I'm having an issue with removing the field named "type".

I have created an index template for my data and declare a type "MyType".

Ingesting data with logstash works very well, but I get an extra field named "type" with the value "MyType" which I did not want (I do not mean the field "_type", which I can make disappear by changing Kibana settings). Is this field generated by Logstash or by Elasticsearch?

When I remove that field in my logstash conf, it actually disappears, but
a mysterious mapping type named "logs" is generated when I check the index mappings. Also the value "logs" is written to the "_type" field.

Is this intended behaviour?

Thanks for any clues.
Yves.

2

Yes, that is the _type - https://www.elastic.co/guide/en/elasticsearch/reference/5.4/_basic_concepts.html#_type

3

Thanks, Mark, I understand the _type field, I am just wondering why the "type" field is created and why, if you remove that field, another mapping named "logs" is created. I don't want a new mapping type.

4

Are you talking about a type field, that you define, and a _type field?

Because they are different things, and I am not sure if that is what you mean or if you mean _type for both and are just dropping the _.

5

Mark,
I am talking about a "type" field without underscore. In my index template there is no "type" field. I just declare the mapping type in that template with the fields that exist in my data. Apparently when you ingest data with logstash, a field named "type" is automatically created. The value of that field is whatever you entered as mapping type.
The annoying part is that this "type" field appears in Kibana, so in the filter part of Logstash I remove it. This works, but when I check the mapping information of the index, a mapping type named "logs" is automatically created with all the fields that were ingested.
In the end this gets my data ingested and I don't see the "type" field anymore, but the value "logs" is written to "_type" (yes, with the underscore) and my original index template is not used.

6

Cool, making sure we are on the same page! :slight_smile:

They are both "types", one is just a reserved and required item in ES, one is simply a tag that LS is adding.

You should see a type and a _type if you are adding the former in LS.

Also - yes this is confusing, it's one of the reasons we are removing _types :slight_smile:

7

Mark, maybe this will clarify more what I mean

this is my index template :

{
  "order": 0,
  "template": "incoming-aa*",
  "settings": {
    "index": {
      "number_of_shards": "10",
      "number_of_replicas": "1",
      "refresh_interval": "5s"
    }
  },
  "mappings": {
    "incoming-aa": {
      "properties": {
        "source_in": {
          "type": "text"
        },
        "source_location": {
          "type": "geo_point"
        },
        "destination_out": {
          "type": "text"
        }
      }
    }
  },
  "aliases": {
    "Incoming-ALL": {}
  }
}

This is my logstash conf file :

input {
        file {
                path => "/home/*.csv"
                type => "incoming-aa"
                start_position => "beginning"
        }
}

filter {
        if [type] == "incoming-aa" {
				# write to @metadata so I can use it in the output part
                mutate { add_field => ["[@metadata][type]", "%{type}"] }

                # split the lines
				csv {
                        separator => ","
                        columns => ["source_in","source_location","destination_out"]
                }

                # remove unnecessary fields
                mutate {
                        remove_field => ["@version", "host", "path", "message", "type"]
                }
				
output {
        if [@metadata][type] == "incoming-aa" {
            elasticsearch {
                hosts => ["10.100.200.100"]
                action => "index"
                index => "incoming-aa-%{+YYYY_MM}"
            	}
	}
}

After ingestion, I check the mapping for that index and below is the result. The part starting and ending with "**" (I have put that there so you can see it better, in reality the asterisks are not there) was added by LS/ES, so the original mapping I set up in the index template was not used. An automatic mapping was done and it named it "logs".
This only happens when I remove the "type" field ! When I do not remove it, my mapping is used, but then the "type" field appears in Kibana and I don't want that.

Edit : Title of this discussion should change to "ELK 5.4 - LS reverts to automatic mapping when 'type' field is removed or renamed"

{
  "incoming-aa-2017_06": {
    "mappings": {
      "incoming-aa": {
        "properties": {
          "source_in": {
            "type": "text"
          },
          "source_location": {
            "type": "geo_point"
          },
          "destination_out": {
            "type": "text"
          }
        }
      },
**"logs": {**
**            "properties": {**
**              "@timestamp": {**
**                "type": "date"**
**              },**
**              "source_in": {**
**                "type": "text",**
**                "fields": {**
**                  "keyword": {**
**                    "type": "keyword",**
**                    "ignore_above": 256**
**                  }**
**                }**
**              },**
**              "source_location": {**
**                "type": "geo_point"**
**              },**
**              "destination_out": {**
**                "type": "text",**
**                "fields": {**
**                  "keyword": {**
**                    "type": "keyword",**
**                    "ignore_above": 256**
**                  }**
**                }**
**              }**
**            }**
**          }**
**        }**
**      }**
}
8

Thanks for the extra info :smiley: [quote="smyttie, post:7, topic:88534"]
"mappings": {
"incoming-aa":
[/quote]

That equates to the index _type.[quote="smyttie, post:7, topic:88534"]
if [type] ==
[/quote]

That's simply a tag that LS can use to do conditionals.[quote="smyttie, post:7, topic:88534"]
"logs": {
[/quote]

That is the document _type that LS uses by default, unless you specify something in the config.

9

You can do that by clicking the spanner beside the subject.

10

I missed that and created a new topic with that title. Thanks for the info. I will remember it for next time.

11

Mark, I am still following... good for me :+1:

But... do you have an explanation to why LS reverts to automatic mapping when I remove the "type" tag? Or even better do you know a way to make that tag disappear in Kibana, without LS reverting to automatic mapping?

12

Where are you setting the type in the LS config? (Cause I cannot see it)

13

I suppose LS gets it from "input > File > Type=..." on top of the conf file.

I remove it just before the "Output" part : remove_field => ["@version", "host", "path", "message", "type"]

14

Ahh yeah, sorry I dunno how I missed that!

Ok, so unless you set https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-document_type, then it'll use the default logs type.

15

Thanks for the tip, Mark. I will try it on Monday. After a week of ELK stacking we have earned a couple of days off, I believe :slight_smile:

16

That helped, thanks Mark. I was confused and did not entirely get the mechanism used by LS. Now I use "document_type" to impose the mapping I set in my indexing template.

Also, I got finally rid of the "type" field by discovering that you can add fields in the "file" plugin :blush: and so @metadata came in handy :

input {
    file {
        ...
        add_field => ["[@metadata][type]", "MyWayOfIndexing"]
    }
}

filter {
    if [@metadata][type] == "MyWayOfIndexing" {
        ...
    }
}

output {
    if [@metadata][type] == "MyWayOfIndexing" {
        elasticsearch {
            host => "TheThing"
            action => "index"
            index => "MyLovelyIndex-%{+YYYY.MM}"
            # I conveniently named the type in my index template 
            # the same as the type field I use in this conf file
            document_type => "MyWayOfIndexing"
        }
    }        
}

Thanks for the help!

17

It's not super clear is it, which is one reason we are ditching types in ES.

18

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.