When I parsed the nginx log using Filebeat pipeline, a user_agent field in the log failed to be parsed. The following error is displayed.
{"type":"mapper_parsing_exception","reason":"object mapping for [user_agent] tried to parse field [user_agent] as object, but found a concrete value
5月 19 17:00:03 k8s-node-01 filebeat[10542]: 2023-05-19T17:00:03.025+0800 WARN [elasticsearch] elasticsearch/client.go:414 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2023, time.May, 19, 17, 0, 2, 574674762, time.Local), Meta:null, Fields:{"agent":{"name":"k8s-node-01"},"ecs":{},"fields":{"log_type":"gate"},"host":{"name":"k8s-node-01"},"input":{},"log":{"file":{"path":"XXXX.log"}},"message":"client_address:XXXX###client_user:-###real_ip:-###visit_time:2023-05-19T15:48:59+08:00###request_uri:POST /xxx/xxx/xxx HTTP/1.1###request_host:xxxx###http_status:200###upstream_status:200###traffic:76###original_address:xxxx###user_agent:Mozilla/5.0 (iPhone; CPU iPhone OS 15_7_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148 MicroMessenger/8.0.29(0x18001d38) NetType/4G Language/zh_CN###request_length:637###load_balancer:172.16.50.94:32007###processing_time:0.044###upstream_response_time:0.044###http_acl_t:-###","tags":["xxx"]}, Private:(*input_logfile.updateOp)(0xc0023c9580), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"object mapping for [user_agent] tried to parse field [user_agent] as object, but found a concrete value"}, dropping event!
- pipeline
POST _ingest/pipeline/_simulate
{
"pipeline": {
"description" : "log_pipeline",
"processors" : [
{
"grok" : {
"field" : "message",
"patterns" : [
"""%{WORD:client_address}:%{IP:client_address}###%{WORD:client_user}:%{DATA:client_user}###%{WORD:real_ip}:%{DATA:real_ip}###%{WORD:visit_time}:%{TIMESTAMP_ISO8601:visit_time}###%{WORD:request_uri}:%{GREEDYDATA:request_uri}###%{WORD:request_host}:%{DATA:request_host}###%{WORD:http_status}:%{NUMBER:http_status}###%{WORD:upstream_status}:%{NUMBER:upstream_status}###%{WORD:traffic}:%{NUMBER:traffic}###%{WORD:original_address}:%{DATA:original_address}###%{WORD:user_agent}:%{GREEDYDATA:user_agent}###%{WORD:request_length}:%{NUMBER:request_length}###%{WORD:load_balancer}:%{DATA:load_balancer}###%{WORD:processing_time}:%{NUMBER:processing_time}###%{WORD:upstream_response_time}:%{NUMBER:upstream_response_time}###%{WORD:http_acl_t}:%{DATA:http_acl_t}###"""
],
"ignore_failure" : true
}
},
{
"date" : {
"field" : "visit_time",
"target_field" : "@timestamp",
"formats" : ["ISO8601"],
"timezone" : "Asia/Shanghai"
}
},
{
"geoip": {
"field": "client_address",
"ignore_failure": true
}
}
]
},
"docs": [
{
"_source": {
"message": """client_address:XXXX###client_user:-###real_ip:-###visit_time:2023-05-19T15:48:59+08:00###request_uri:POST /xxx/xxx/xxx HTTP/1.1###request_host:xxxx###http_status:200###upstream_status:200###traffic:76###original_address:xxxx###user_agent:Mozilla/5.0 (iPhone; CPU iPhone OS 15_7_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148 MicroMessenger/8.0.29(0x18001d38) NetType/4G Language/zh_CN###request_length:637###load_balancer:172.16.50.94:32007###processing_time:0.044###upstream_response_time:0.044###http_acl_t:-###"""
}
}
]
}
However, the kibana console tested the pipeline without any problems, it was parsed successfully, and started filebeat parsing with an error
