Filebeat.yml include_lines and decode_json can't work together

Hi:
Can anyone help me ?
I meet problem using filebeat, error message:
Exiting: Error in initing prospector: When using the JSON decoder and line filtering together, you need to specify a message_key value accessing 'filebeat.prospectors.0' (source:'filebeat.yml')
Exiting: Error in initing prospector: When using the JSON decoder and line filtering together, you need to specify a message_key value accessing 'filebeat.prospectors.0' (source:'filebeat.yml')

----------------------------------------------filebeat.yml----------------------------------------------------------------
filebeat.prospectors:

- type: log
  enabled: true
  json.keys_under_root: true
  paths:
    - /data/weblog/nginx/nginx.log
	
  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  include_lines: ['/api/datasources/proxy/']

processors:
    - decode_json_fields:
        fields: ["message"]
        target: json
        max_depth: 1

----------------------------------------------filebeat debug message----------------------------------------
2018-05-11T10:58:43.129+0800 DEBUG [publish] pipeline/processor.go:275 Publish event: {
"@timestamp": "2018-05-11T02:58:43.129Z",
"@metadata": {
"beat": "filebeat",
"type": "doc",
"version": "6.2.4"
},
"source": "/data/weblog/nginx/nginx.log",
"beat": {
"version": "6.2.4",
"name": "8.ops.com",
"hostname": "8.ops.com"
},
"method": "GET",
"time": "11/May/2018:09:36:17 +0800",
"cost": "0.010",
"prospector": {
"type": "log"
},
"body": "",
"status": "200",
"uri": "GET /d/000000321/neng?refresh=1m\u0026orgId=43 HTTP/1.1",
"offset": 1780916
}

ths ahead.

I can see how the log message might be confusing here. It actually does not refer to the processor but to json.* config inside the prospector. You have to set json.message_key: https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html#config-json

As I see you are using the processor and the json option I wonder if that is really what you want? There are use cases for decoded json in json but normally that is not needed.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.