Hey @markov00. Thanks for the follow up, and the patience!
I finally realised that logstash config is stored locally on our ELK box (rather than in ES somewhere) and in the /etc/logstash/conf.d directory I see 3 files:
02-beats-input.conf:
input {
beats {
port => 5044
codec => "json"
ssl => true
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}
10-syslog-filter.conf: this one just applies a filter for syslog type records, so skipping this.
30-elasticsearch-output.conf:
output {
elasticsearch {
hosts => ["localhost:9200"]
sniffing => true
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
What is meaning of the json codec? Does this magically create and update all the required ES index mappings? (Update: seems unlikely...)
So is that logstash setup basically useless? I should ignore logstash altogether?
One thing I'm still confused about is where the mapping is happening on our current 6.x box.
I dumped all the index templates from our 6.x box using GET _template and I can't see any mention of a json or msg prop. Yet, the result of say GET filebeat-2020.03.22/_mapping shows a mapping for them:
{
"filebeat-2020.03.22" : {
"mappings" : {
"doc" : {
"properties" : {
"@timestamp" : {
"type" : "date"
},
....
"json" : {
"properties" : {
"err" : {
"properties" : {
"code" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"message" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"stack" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"hostname" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"level" : {
"type" : "long"
},
"levelStr" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"msg" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"pid" : {
"type" : "long"
},
"time" : {
"type" : "date"
},
"v" : {
"type" : "long"
}
}
},
....
And looking at the index patterns from GET _cat/templates?v:
name index_patterns order version
kibana_index_template:.kibana [.kibana] 0
.ml-state [.ml-state*] 0 6081099
.monitoring-es [.monitoring-es-6-*] 0 6070299
logstash-index-template [.logstash] 0
.ml-meta [.ml-meta] 0 6081099
.ml-notifications [.ml-notifications] 0 6081099
.ml-anomalies- [.ml-anomalies-*] 0 6081099
.monitoring-beats [.monitoring-beats-6-*] 0 6070299
.management-beats [.management-beats] 0 67000
filebeat-6.0.0 [filebeat-6.0.0-*] 1
.monitoring-logstash [.monitoring-logstash-6-*] 0 6070299
.ml-config [.ml-config] 0 6081099
.watches [.watches*] 2147483647
security-index-template [.security-*] 1000
security_audit_log [.security_audit_log*] 1000
.monitoring-kibana [.monitoring-kibana-6-*] 0 6070299
.triggered_watches [.triggered_watches*] 2147483647
.watch-history-9 [.watcher-history-9*] 2147483647
.monitoring-alerts [.monitoring-alerts-6] 0 6070299
.kibana_task_manager [.kibana_task_manager] 0 6081099
It doesn't seem like there is an index pattern that actually matches an index name like filebeat-2020.03.22! So how does it know about the json prop mappings? Does it just fallback to looking at the previous index? If so, I guess I must have followed some guide to configure an initial index correctly at some point.
Sorry for all the questions, I just really want to understand how it all plugs together before I proceed.
Thanks!
