Filebeat include_lines and decode_json can't work

Elastic Stack Beats
1

Hi

I meet problem:

"Exiting: Error in initing prospector: When using the JSON decoder and line filtering together, you need to specify a message_key value accessing 'filebeat.prospectors.0' (source:'filebeat.yml')
Exiting: Error in initing prospector: When using the JSON decoder and line filtering together, you need to specify a message_key value accessing 'filebeat.prospectors.0' (source:'filebeat.yml')"

my **filebeat.yml**
----------------------------------------------------------------------------------------------------------------------------------
   filebeat.prospectors:
- type: log
  enabled: true
  json.keys_under_root: true
  paths:
    - /data/weblog/nginx/nginx.log
	
  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  include_lines: ['/api/datasources/proxy/']

processors:
    - decode_json_fields:
        fields: ["message"]
        target: json
        max_depth: 1

filebeat debug log:

2018-05-11T10:58:43.129+0800	DEBUG	[publish]	pipeline/processor.go:275	Publish event: {
  "@timestamp": "2018-05-11T02:58:43.129Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.2.4"
  },
  "source": "/data/weblog/nginx/nginx.log",
  "beat": {
    "version": "6.2.4",
    "name": "8.ops.com",
    "hostname": "8.ops.com"
  },
  "method": "GET",
  "time": "11/May/2018:09:36:17 +0800",
  "cost": "0.010",
  "prospector": {
    "type": "log"
  },
  "body": "",
  "status": "200",
  "uri": "GET /d/000000321/neng?refresh=1m\u0026orgId=43 HTTP/1.1",
  "offset": 1780916
}

Can anyone help me.

ths

2

Line filtering in Beats is done based on the content read from the input. In case of JSON that content is empty, because all of the key value pairs of the JSON is added to the fields of the message. So there is nothing which can be matched. To avoid that you need to select a field from the incoming JSON which ca be matched against the pattern.

In your case, if I understood correctly your config, you would want to match the value of uri in your message. The following config should work in that case:

include_lines: ['/api/datasources/proxy/']

# Decode JSON options. Enable this if your logs are structured in JSON.
# JSON key on which to apply the line filtering and multiline settings. This key
# must be top level and its value must be string, otherwise it is ignored. If
# no text key is defined, the line filtering and multiline features cannot be used.
json.message_key: uri
2 Likes
3

thanks very much .

It works

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.