Index Templates - Help Needed

Elastic Stack Elasticsearch
1

I am trying to get a more refined template for my indexes in place based on my data so numbers are numbers/text is text/etc/etc.

I am using logstash to put everything into elasticsearch so it is going into logstash-applicationlogs--YMD, I have logstash managing the index for logstash-* which gets me some basic points like geoip/etc.

But when i try and add another template with a higher order (1,10,100/etc) it doesn't seem to have any affect, in essence the fields i specify in my template to be numbers, are still strings. I have tried being less specific on the index (logstash-* vs logstash-applicationlogs-testapp-*). Each time I completely destroy the current dates index so new data can be indexed using the template and nothing changes.

Here is my template:

{
	"template" : "logstash-applicationlogs-testapp-*",
	"order": 10,
	"settings" : {
		"index" : {
			"refresh_interval" : "5s"
		}
	},
	"mappings": {
		"log": {
			"_source": { "enabled": true },
			"properties": {
				"request": { "type": "text", "index": "not_analyzed"},
				"agent": { "type": "text", "index": "not_analyzed"},
				"auth": { "type": "text", "index": "not_analyzed"},
				"bytes": { "type": "integer", "index": "not_analyzed"},
				"clientip": { "type": "ip", "index": "not_analyzed"},
				"host": { "type": "text", "index": "not_analyzed"},
				"httpversion": { "type": "text", "index": "not_analyzed"},
				"ident": { "type": "text", "index": "not_analyzed"},
				"logstash_host": { "type": "text", "index": "not_analyzed"},
				"message": { "type": "text", "index": "not_analyzed"},
				"referrer": { "type": "text", "index": "not_analyzed"},
				"response": { "type": "integer", "index": "not_analyzed"},
				"source": { "type": "text", "index": "not_analyzed"},
				"verb": { "type": "text", "index": "not_analyzed"},
				"timestamp": { "type": "date", "format": "dd/MMM/yyyy:HH:mm:ss Z" }
			}
		}
	}
}

here is an example entry I am putting in (apache logs)

{
  "_index": "logstash-applicationlogs-testapp-2016.11.14",
  "_type": "log",
  "_id": "AVhk_TBhw7701Hi_Z8lv",
  "_score": null,
  "_source": {
    "request": "/testAny.pl?uri=/status&srch=Success&restart=1&send=0",
    "agent": "\"-\"",
    "geoip": {},
    "offset": 3695652,
    "auth": "-",
    "logstash_host": "ftxlog01",
    "ident": "-",
    "input_type": "log",
    "verb": "GET",
    "source": "/opt/Logs/FLT-access.txt",
    "message": "10.150.8.3 - - [14/Nov/2016:14:37:31 -0800] \"GET /testAny.pl?uri=/status&srch=Success&restart=1&send=0\" 200 103 \"-\" \"-\"",
    "type": "log",
    "tags": [
      "apachelog",
      "fleet",
      "beats_input_codec_plain_applied",
      "_geoip_lookup_failure"
    ],
    "referrer": "\"-\"",
    "@timestamp": "2016-11-14T22:37:38.003Z",
    "response": "200",
    "bytes": "103",
    "clientip": "10.150.8.3",
    "@version": "1",
    "beat": {
      "hostname": "ftxvmapp09",
      "name": "ftxvmapp09",
      "version": "5.0.0"
    },
    "host": "ftxvmapp09",
    "timestamp": "14/Nov/2016:14:37:31 -0800"
  },
  "fields": {
    "@timestamp": [
      1479163058003
    ]
  },
  "sort": [
    1479163058003
  ]
}

I am trying to make it so things like bytes and response code are numbers so i can perform math on them, but they never seem to swap which to me says the template I built isn't getting applied as I did something wrong or am just missing something entirely.

Edit: Just realized I didn't say what version, running 5.0.0

2

Highest order is 0, so perhaps that is the problem?

3

If that is the case, then the documentation is completely wrong:

The order of the merging can be controlled using the order parameter, with lower order being applied first, and higher orders overriding them.

From: https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html#multiple-templates

4

No, it's just confusing. Even to me....

0 is the priority that gets applied first.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.