# Filebeat include\_lines and decode\_json can't work

**URL:** <https://discuss.elastic.co/t/filebeat-include-lines-and-decode-json-cant-work/131383>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [May 11, 2018, 3:18am UTC](https://discuss.elastic.co/t/filebeat-include-lines-and-decode-json-cant-work/131383 "2018-05-11T03:18:21Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Qing](https://avatars.discourse-cdn.com/v4/letter/q/e99b99/32.png) [@Qing](https://discuss.elastic.co/u/Qing)\
**Post date:** [May 11, 2018, 3:18am UTC](https://discuss.elastic.co/t/filebeat-include-lines-and-decode-json-cant-work/131383/1 "2018-05-11T03:18:21Z")

</div>

Hi

I meet problem:

"Exiting: Error in initing prospector: When using the JSON decoder and line filtering together, you need to specify a message\_key value accessing 'filebeat.prospectors.0' (source:'filebeat.yml')  
Exiting: Error in initing prospector: When using the JSON decoder and line filtering together, you need to specify a message\_key value accessing 'filebeat.prospectors.0' (source:'filebeat.yml')"

```
my **filebeat.yml**
----------------------------------------------------------------------------------------------------------------------------------
   filebeat.prospectors:
- type: log
  enabled: true
  json.keys_under_root: true
  paths:
    - /data/weblog/nginx/nginx.log
	
  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  include_lines: ['/api/datasources/proxy/']

processors:
    - decode_json_fields:
        fields: ["message"]
        target: json
        max_depth: 1

```

* * *

**filebeat debug log:**

```
2018-05-11T10:58:43.129+0800	DEBUG	[publish]	pipeline/processor.go:275	Publish event: {
  "@timestamp": "2018-05-11T02:58:43.129Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.2.4"
  },
  "source": "/data/weblog/nginx/nginx.log",
  "beat": {
    "version": "6.2.4",
    "name": "8.ops.com",
    "hostname": "8.ops.com"
  },
  "method": "GET",
  "time": "11/May/2018:09:36:17 +0800",
  "cost": "0.010",
  "prospector": {
    "type": "log"
  },
  "body": "",
  "status": "200",
  "uri": "GET /d/000000321/neng?refresh=1m\u0026orgId=43 HTTP/1.1",
  "offset": 1780916
}

```

Can anyone help me.

ths

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [May 11, 2018, 6:48am UTC](https://discuss.elastic.co/t/filebeat-include-lines-and-decode-json-cant-work/131383/2 "2018-05-11T06:48:29Z")

</div>

Line filtering in Beats is done based on the content read from the input. In case of JSON that content is empty, because all of the key value pairs of the JSON is added to the fields of the message. So there is nothing which can be matched. To avoid that you need to select a field from the incoming JSON which ca be matched against the pattern.

In your case, if I understood correctly your config, you would want to match the value of `uri` in your message. The following config should work in that case:

```auto
include_lines: ['/api/datasources/proxy/']

# Decode JSON options. Enable this if your logs are structured in JSON.
# JSON key on which to apply the line filtering and multiline settings. This key
# must be top level and its value must be string, otherwise it is ignored. If
# no text key is defined, the line filtering and multiline features cannot be used.
json.message_key: uri

```

---

<div class="post-metadata">

**Author:** ![Qing](https://avatars.discourse-cdn.com/v4/letter/q/e99b99/32.png) [@Qing](https://discuss.elastic.co/u/Qing)\
**Post date:** [May 11, 2018, 7:44am UTC](https://discuss.elastic.co/t/filebeat-include-lines-and-decode-json-cant-work/131383/3 "2018-05-11T07:44:53Z")

</div>

> [@kvch](#):
>
> the

thanks very much .

It works

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 8, 2018, 7:44am UTC](https://discuss.elastic.co/t/filebeat-include-lines-and-decode-json-cant-work/131383/4 "2018-06-08T07:44:54Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.