I used logstash to parse the JSON log, but the ES index did not generate the corresponding fields
logstash config:
filter {
if [service] == "roomtest" {
grok {
match => { "message" => "(?<timestamp>\d{4}/\d{2}/\d{2}\s\d{2}:\d{2}:\d{2}.\d{6}) (?<info>(\w{4})),(?<jsonmsg>([\s\S]*))" }
}
json {
skip_on_invalid_json => true
source => "jsonmsg"
}
mutate {
remove_field => ["message"]
}
} else {
json {
skip_on_invalid_json => true
source => "message"
}
}
}
logstash Grok results:
"@version" => "1",
"timestamp" => "2021/07/27 23:50:56.730053",
"PlayerId" => 303792217,
"host" => "logstash151",
"Outline" => {
"价值消耗" => 0,
"价值峰值" => 15105589,
"价值峰值时间" => "2021-07-27 23:50:54",
"money快照" => {
"leave-value" => 15105589,
"login-value" => 15105589,
"leave-money" => 14683689,
"login-time" => "2021-07-27 23:50:54",
"leave-time" => "2021-07-27 23:50:56",
"login-money" => 14683689
},
"金币峰值" => 14683689,
"金币峰值时间" => "2021-07-27 23:50:54",
"初始价值" => 15105589,
"金币获得" => 0,
"最大赢分" => "",
"初始金币" => 14683689,
"金币消耗" => 0,
"价值获得" => 0
},
"MoneyDetail" => "",
"MoneySum" => "",
"@timestamp" => 2021-08-09T07:53:24.865Z,
"Traces" => [],
"jsonmsg" => "{\"PlayerId\":303792217,\"Outline\":{\"初始金币\":14683689,\"金币消耗\":0,\"金币获得\":0,\"金币峰值\":14683689,\"金币峰值时间\":\"2021-07-27 23:50:54\",\"最大赢分\":\"\",\"初始价值\":15105589,\"价值消耗\":0,\"价值获得\":0,\"价值峰值\":15105589,\"价值峰值时间\":\"2021-07-27 23:50:54\",\"money快照\":{\"login-money\":14683689,\"login-value\":15105589,\"login-time\":\"2021-07-27 23:50:54\",\"leave-money\":14683689,\"leave-value\":15105589,\"leave-time\":\"2021-07-27 23:50:56\"}},
es template:
{
"dynamic_templates": [
{
"message_field": {
"mapping": {
"fielddata": {
"format": "disabled"
},
"index": "analyzed",
"omit_norms": true,
"type": "string"
},
"match_mapping_type": "string",
"match": "message"
}
},
{
"string_fields": {
"mapping": {
"index": "not_analyzed",
"type": "string",
"doc_values": true
},
"match_mapping_type": "string",
"match": "*"
}
}
],
"date_detection": false,
"properties": {
"@timestamp": {
"type": "date"
},
"ecs": {
"type": "object",
"properties": {
"version": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"data_stream": {
"type": "object",
"properties": {
"namespace": {
"type": "constant_keyword"
},
"type": {
"type": "constant_keyword",
"value": "logs"
},
"dataset": {
"type": "constant_keyword"
}
}
},
"money快照": {
"type": "nested",
"properties": {
"login-value": {
"type": "integer"
}
}
},
"host": {
"type": "object",
"properties": {
"ip": {
"type": "ip"
}
}
},
"jsonmsg": {
"eager_global_ordinals": false,
"index_phrases": false,
"fielddata": false,
"norms": true,
"index": true,
"store": false,
"type": "text",
"index_options": "positions"
}
}
}
Some new settings of manual support:
"money快照": {
"type": "nested",
"properties": {
"login-value": {
"type": "integer"
}
}
How to make the ES template generate the fields parsed by logstash