Logstash/Elasticseach and template issue

[Delphin]

Hello,

I am using ELK 6.5.3 with Docker-Compose.

I have an issue between the index template defined at logstash-side and the indexed documents.

logstash.config

input {
	beats {
		host => "0.0.0.0"
		port => 5000
	}
}
filter {
	[...]
}
output {
	elasticsearch {
		id => "es-monitoring"
		hosts => "elasticsearch:9200"
		index => "logs-%{+YYYY.MM.dd}"
		template => "/usr/share/logstash/config/logs-template.json"
		template_name => "logs"
		template_overwrite => true
	}
 	stdout { codec => rubydebug	}
}

logs-template.json

 {
  "index_patterns": ["logs-*"],
  "settings": {
    "number_of_shards": 1,
	"number_of_replicas" : 0
  },
  "mappings" : {
    "doc" : {
      ...
      "properties" : {
        "response" : {
          "type" : "short"
        },
        ...
      }
    }
  }
}

Using the Elasticsearch API, I can see the template:

https://localhost:9200/_template/logs/?pretty

{
  "logs" : {
    "order" : 0,
    "index_patterns" : [
      "logs-*"
    ],
    "settings" : {
      "index" : {
        "number_of_shards" : "1",
        "number_of_replicas" : "0"
      }
    },
    "mappings" : {
      ...
      "response" : {
        "type" : "short"
      }
      ...
    }
  }
}

When the first document is sent from logstash to elasticsearch:

[2018-12-19T13:19:46,255][INFO ][o.e.c.m.MetaDataIndexTemplateService] [o4hWvG8] adding template [logs] for index patterns [logs-*]

Under Kibana (Index Patterns), the response field is well defined:

Type = number

But when having a look to the document at Elasticsearch-side, the property is string:

"response" : "200"

What's happening?
Thanks.

[warkolm (Mark Walkom)]

What's the actual mapping for the field show as?

[Delphin]

Thanks warkolm for your question.

I don't what's happened but now it seems to be OK :\

[Delphin]

It is OK because of the logstash pipeline:

mutate {
  convert => [ "response", "integer" ]
  convert => [ "bytes", "integer" ]
}

Template:

"nginx-logs" : {
  "order" : 0,
  "index_patterns" : [
    "nginx-logs-*"
  ],
  "settings" : {
    "index" : {
      "number_of_shards" : "1",
      "number_of_replicas" : "0"
    }
  },
  "mappings" : {
    "doc" : {
      "properties" : {
        "bytes" : {
          "type" : "short"
        },
        "response" : {
          "type" : "long"
        }
      }
    }
  },
  "aliases" : { }
}

If I remove the mutate { } block then the first indexed document looks like this:

{
  "_index": "nginx-logs-2018.12.22",
  "_type": "doc",
  "_id": "BWMQ2GcBSowEK9_MFsY_",
  "_version": 1,
  "_score": null,
  "_source": {
    "bytes": "104",
    "response": "200",
  }
}

[Delphin]

Strange...

mutate {
  convert => [ "response", "integer" ] # OK
  convert => [ "bytes", "integer" ] # OK
}

mutate {
  add_field => { "[browser][version_major]" => "%{major}" }
  add_field => { "[browser][version_minor]" => "%{minor}" }
  add_field => { "[browser][version_patch]" => "%{patch}" }

  convert => [ "[browser][version_major]", "integer" ] # KO
  convert => [ "[browser][version_minor]", "integer" ] # KO
  convert => [ "[browser][version_patch]", "integer" ] # KO
}

Template:

{
  "index_patterns": ["nginx-logs-*"],
  "settings": {
    "number_of_shards": 1,
	"number_of_replicas" : 0
  },
  "mappings" : {
    "doc" : {
      "properties" : {
        "bytes" : {
          "type" : "integer" <-- only works using convert { }
        },
        "browser" : {
          "properties" : {
            "version_major" : { <-- KO
              "type" : "integer"
            },
            "version_minor" : { <-- KO
              "type" : "integer"
            },
            "version_patch" : { <-- KO
              "type" : "integer"
            }
          }
        },
        "response" : {
          "type" : "integer" <-- only works using convert { }
        }
      }
    }
  }
}

Indexed document:

"response": 200,
"bytes": 4089,
"browser": {
  "version_major": "57"
  "version_minor": "0",
  "version_patch": "3098",
}