Template not being applied to elasticsearch output

Elastic Stack Logstash
1

In Nginx, I've converted my log_format to json and sending the logs via Filebeat to logstash. I'm trying to set some of my fields to not_analyzed but nothing I'm doing is working. Almost all the fields from nginx logs are being set to analzyed. I'm testing out ELK so I've been repeatedly removing all the indexes and starting from scratch.

Maybe I'm misunderstanding the logstash output template.

My logstash conf:

elasticsearch {
      hosts => ["xx.xx.xx.xx:9200"]
      sniffing => true
      manage_template => false
      index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
      document_type => "%{[metadata][type]}"
      template_name => "filebeat-*"
      template => "/etc/logstash/mappings/filebeat.json"
    }

I'm using the default json template that comes with filebeat.

{
  "mappings": {
    "_default_": {
      "_all": {
        "enabled": true,
        "norms": {
          "enabled": false
        }
      },
      "dynamic_templates": [
        {
          "template1": {
            "mapping": {
              "doc_values": true,
              "ignore_above": 1024,
              "index": "not_analyzed",
              "type": "{dynamic_type}"
            },
            "match": "*"
          }
        }
      ],
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "message": {
          "type": "string",
          "index": "analyzed"
        },
        "offset": {
          "type": "long",
          "doc_values": "true"
        }
      }
    }
  },
  "settings": {
    "index.refresh_interval": "5s"
  },
  "template": "filebeat-*"
}
2

What's the mapping for the data in that index actually look like?

3

Disabling manage_template but keeping template_name and template set doesn't make sense. With manage_template disabled it's your responsibility to manage the templates. Have you done that? What's the actual contents of the index template according to ES? What's the actual mapping of newly created indexes?

4

@warkolm below is when I run curl http://localhost:9200/filebeat-*

{
  "filebeat-2016.04.13" : {
    "aliases" : { },
    "mappings" : {
      "%{[metadata][type]}" : {
        "properties" : {
          "@timestamp" : {
            "type" : "date",
            "format" : "strict_date_optional_time||epoch_millis"
          },
          "@version" : {
            "type" : "string"
          },
          "beat" : {
            "properties" : {
              "hostname" : {
                "type" : "string"
              },
              "name" : {
                "type" : "string"
              }
            }
          },
          "body_bytes_sent" : {
            "type" : "string"
          },
          "count" : {
            "type" : "long"
          },
          "host" : {
            "type" : "string"
          },
          "http_referrer" : {
            "type" : "string"
          },
          "http_user_agent" : {
            "type" : "string"
          },
          "input_type" : {
            "type" : "string"
          },
          "offset" : {
            "type" : "long"
          },
          "remote_addr" : {
            "type" : "string"
          },
          "remote_user" : {
            "type" : "string"
          },
          "request" : {
            "type" : "string"
          },
          "request_method" : {
            "type" : "string"
          },
          "request_time" : {
            "type" : "string"
          },
          "source" : {
            "type" : "string"
          },
          "status" : {
            "type" : "string"
          },
          "tags" : {
            "type" : "string"
          },
          "time" : {
            "type" : "date",
            "format" : "strict_date_optional_time||epoch_millis"
          },
          "type" : {
            "type" : "string"
          }
        }
      }
    },
    "settings" : {
      "index" : {
        "creation_date" : "1460579107855",
        "uuid" : "BbZP1kmtTzmFp2UZmX85sQ",
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "version" : {
          "created" : "2030199"
        }
      }
    },
    "warmers" : { }
  },
  "filebeat-2016.04.14" : {
    "aliases" : { },
    "mappings" : {
      "%{[metadata][type]}" : {
        "properties" : {
          "@timestamp" : {
            "type" : "date",
            "format" : "strict_date_optional_time||epoch_millis"
          },
          "@version" : {
            "type" : "string"
          },
          "beat" : {
            "properties" : {
              "hostname" : {
                "type" : "string"
              },
              "name" : {
                "type" : "string"
              }
            }
          },
          "body_bytes_sent" : {
            "type" : "string"
          },
          "count" : {
            "type" : "long"
          },
          "host" : {
            "type" : "string"
          },
          "http_referrer" : {
            "type" : "string"
          },
          "http_user_agent" : {
            "type" : "string"
          },
          "input_type" : {
            "type" : "string"
          },
          "offset" : {
            "type" : "long"
          },
          "remote_addr" : {
            "type" : "string"
          },
          "request" : {
            "type" : "string"
          },
          "request_method" : {
            "type" : "string"
          },
          "request_time" : {
            "type" : "string"
          },
          "source" : {
            "type" : "string"
          },
          "status" : {
            "type" : "string"
          },
          "time" : {
            "type" : "date",
            "format" : "strict_date_optional_time||epoch_millis"
          },
          "type" : {
            "type" : "string"
          }
        }
      }
    },
    "settings" : {
      "index" : {
        "creation_date" : "1460643414579",
        "uuid" : "QFQGsAN8TIWPOriGNqkCRw",
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "version" : {
          "created" : "2030199"
        }
      }
    },
    "warmers" : { }
  }
}

Is that what you're asking for?

5

I was following some examples and was messing around with the settings. I thought setting a template in the elasticsearch output would set the mappings of the index. If I want to apply mappings, can I only do that through the HTTP API?

6

I thought setting a template in the elasticsearch output would set the mappings of the index.

Yes, but not with manage_template disabled.

If I want to apply mappings, can I only do that through the HTTP API?

Well, you could use the Java API but the REST API is certainly more convenient.

Be systematic.

  1. Decide whether you should maintain the index templates yourself or use Logstash.
  2. Regardless of the answer to the previous question, make sure you reach a state where you can verify that the correct template is in place.
  3. Verify that the template is applied when you create an index via the REST API.
  4. Verify that it works when Logstash creates the index.