Filebeat.yml include_lines and decode_json can't work together


(Qing) #1

Hi:
Can anyone help me ?
I meet problem using filebeat, error message:
Exiting: Error in initing prospector: When using the JSON decoder and line filtering together, you need to specify a message_key value accessing 'filebeat.prospectors.0' (source:'filebeat.yml')
Exiting: Error in initing prospector: When using the JSON decoder and line filtering together, you need to specify a message_key value accessing 'filebeat.prospectors.0' (source:'filebeat.yml')

----------------------------------------------filebeat.yml----------------------------------------------------------------
filebeat.prospectors:

- type: log
  enabled: true
  json.keys_under_root: true
  paths:
    - /data/weblog/nginx/nginx.log
	
  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  include_lines: ['/api/datasources/proxy/']

processors:
    - decode_json_fields:
        fields: ["message"]
        target: json
        max_depth: 1

----------------------------------------------filebeat debug message----------------------------------------
2018-05-11T10:58:43.129+0800 DEBUG [publish] pipeline/processor.go:275 Publish event: {
"@timestamp": "2018-05-11T02:58:43.129Z",
"@metadata": {
"beat": "filebeat",
"type": "doc",
"version": "6.2.4"
},
"source": "/data/weblog/nginx/nginx.log",
"beat": {
"version": "6.2.4",
"name": "8.ops.com",
"hostname": "8.ops.com"
},
"method": "GET",
"time": "11/May/2018:09:36:17 +0800",
"cost": "0.010",
"prospector": {
"type": "log"
},
"body": "",
"status": "200",
"uri": "GET /d/000000321/neng?refresh=1m\u0026orgId=43 HTTP/1.1",
"offset": 1780916
}

ths ahead.


(ruflin) #2

I can see how the log message might be confusing here. It actually does not refer to the processor but to json.* config inside the prospector. You have to set json.message_key: https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html#config-json

As I see you are using the processor and the json option I wonder if that is really what you want? There are use cases for decoded json in json but normally that is not needed.


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.