ES 6.x: mapping types


(Jeroen Ruigrok Van Der Werven) #1

Currently in the process of upgrading our Elastic setup to 6.x and I am a bit lost with the intent of some changes made.

In the past I had set up Logstash to accept apache log files via filebeat as well as a manual spooling process via the input plugin. With the input plugin I had things like:

input {
  file {
    path => "/path/to/site_access.log"
    start_position => "beginning"
    "[@metadata][type]" => "apache"
  }
}

filter {
  if [@metadata][type] == "apache" {
    mutate {
      replace => {
        "host" => "indexer01"
      }
    }
  }
}

And then the apache filter:

filter {
  if [@metadata][type] == "apache" {
    grok {
      match => { "message" => "\A\[%{HTTPDATE:accept_date}\] %{IP:client_ip} %{NOTSPACE:tls_protocol} %{NOTSPACE:cipher} \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:http_version})?|%{DATA:raw_request})\" %{NUMBER:status_code:int} %{NOTSPACE:bytes:int} %{NUMBER:duration:int} %{QS:referrer} %{QS:agent}" }
    }
    date {
      match => [ "accept_date", "dd/MMM/yyyy:HH:mm:ss Z" ]
    }
    mutate {
      gsub => [
        "agent", "\"", "",
        "referrer", "\"", ""
      ]
    }
    ruby {
      code => "
        begin
          event.set('bytes', nil) if event.get('bytes') == '-'
          event.set('duration', nil) if event.get('duration') == '-'
          event.set('agent', nil) if event.get('agent') == '-'
          event.set('referrer', nil) if event.get('referrer') == '-'
        end
      "
    }
  }
}

Followed by elasticsearch output:

output {
  elasticsearch {
    hosts => ["http://storage01:9200"]
    index => "%{[@metadata][type]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

And on Elasticsearch I have a template:

{
  "index_patterns": ["apache-*"],
  "mappings": {
    "apache": {
      "properties": {
        "accept_date": {
          "type": "date",
          "format": "dd/MMM/yyyy:HH:mm:ss Z||strict_date_optional_time||epoch_millis"
        },
        "agent": {
          "type": "keyword"
        },
        "bytes": {
          "type": "long"
        },
        "cipher": {
          "type": "keyword"
        },
        "client_ip": {
          "type": "ip"
        },
        "duration": {
          "type": "long"
        },
        "host": {
          "type": "keyword"
        },
        "http_version": {
          "type": "keyword"
        },
        "referrer": {
          "type": "text"
        },
        "request": {
          "type": "text"
        },
        "status_code": {
          "type": "keyword"
        },
        "tls_protocol": {
          "type": "keyword"
        },
        "verb": {
          "type": "keyword"
        }
      }
    }
  }
}

This will result in data in Elasticsearch with _type set to apache.

Now, the moment I remove document_type, since it's deprecated, the elasticsearch output plugin will set _type to doc (as documented on document_type). This will result in an error: Rejecting mapping update to [apache-2018.02.08] as the final mapping would have more than 1 type: [apache, doc]

What is the intended way forward? I can work around it by specifying the mapping name in the apache template as doc, but that seems a bit silly. I can also keep document_type around to set it to apache, which will then match the template's mapping name, but that would sort of defeat the whole purpose of removing deprecated things.


(Aaron Mildenstein) #2

The best way forward is to just create a new index with a new template that uses doc instead of apache.

You might need to delete the current apache-2018.02.08 for this to work. Or, you could stop ingestion, reindex that index to apache-2018.02.08_1 with doc as the mapping type, and then delete the old apache-2018.02.08 and let Logstash create a "new" version with doc as the mapping type, preserving your index data.


(Jeroen Ruigrok Van Der Werven) #3

@theuntergeek Alright, then I'll do that. No need for reindexing, thankfully, since I am able to build everything from scratch at the moment.


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.