Hi,
I'm rather new to Elastic and now I'm running into a strange problem I do not understand and did not find the right pointer to fix it.
I think everyone worked fine in the beginning but then I added a new input and output definition to logstash which most likely broke something but to me it does not make much sense.
So the current scenario:
I'm collecting via logstash from two different inputs. One is filebeat and the other http json.
I want both to end up in different indexes and apply different filters.
So I tag the JSON input manually with a certain tag ("wforce"):
input {
http {
port => 8080
codec => json
type => wforce_report
add_field => { "input" => "http" }
tags => ["wforce"]
}
}
input {
beats {
port => 5044
tags => ["beats"]
}
}
I skip the filter definitions since I do not think they are relevant here.
Output definition looks like this:
output {
if "wforce" in [tags] {
elasticsearch {
hosts => "elasticsearch:9200"
index => "logstash-wforce-%{+YYYY.MM.dd}"
template => "/tmp/templates/wforce_template.json"
template_name => "wforce"
template_overwrite => true
user => elastic
password => changeme
}
} else {
elasticsearch {
hosts => "elasticsearch:9200"
index => "logstash-mail-%{+YYYY.MM.dd}"
user => elastic
password => changeme
}
}
}
So the index definition works totally fine but something is strange with the template.
{
"index_patterns" : ["logstash-wforce*"],
"settings" : { "index.refresh_interval" : "5s"},
"mappings" : {
"dynamic_templates" :
[
{
"minor_fields" : {
"match" : "*minor",
"mapping" : { "type" : "integer", "index" : true }
}
},
{
"major_fields" : {
"match" : "*major",
"mapping" : { "type" : "integer", "index" : true }
}
},
{
"string_fields" : {
"match_mapping_type" : "string",
"mapping" : { "type" : "keyword", "index" : true }
}
}
],
"properties" : {
"geoip" : {
"dynamic": true,
"properties" : {
"ip": { "type": "ip" },
"location" : { "type" : "geo_point" },
"latitude" : { "type" : "half_float" },
"longitude" : { "type" : "half_float" }
}
},
"policy_reject": { "type": "boolean" },
"success": { "type": "boolean"},
"tls": { "type": "boolean" },
"t": { "type": "float" }
}
}
}
Now what happens is that I certainly see in elastic in the index management's mapping tab that everything seems to be there. Especially also the minor_fields and major_fields definitions which should make sure these are saved as integers.
But the index pattern fields does not have these as numbers but strings. E.g. the policy_reject boolean works but that just might be because ES recognizes it itself like this.
Any pointer?
Thanks,
Wolfgang