# Ignore lines which are not JSON in log file using Filebeat

**URL:** <https://discuss.elastic.co/t/ignore-lines-which-are-not-json-in-log-file-using-filebeat/187154>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [June 24, 2019, 2:06pm UTC](https://discuss.elastic.co/t/ignore-lines-which-are-not-json-in-log-file-using-filebeat/187154 "2019-06-24T14:06:15Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jitendra\_Kumhar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jitendra_kumhar/32/48640_2.png) [@Jitendra\_Kumhar](https://discuss.elastic.co/u/Jitendra_Kumhar)\
**Post date:** [June 24, 2019, 2:06pm UTC](https://discuss.elastic.co/t/ignore-lines-which-are-not-json-in-log-file-using-filebeat/187154/1 "2019-06-24T14:06:15Z")

</div>

Sample log file

{ "name": "Meowsy", "species" : "cat", "foods": { "likes": ["tuna", "catnip"], "dislikes": ["ham", "zucchini"] } }  
another line  
next line

So I want filebeat to ignore 2nd and 3rd line and send only 1st line  
Any idea how to do that?  
Thank you

---

<div class="post-metadata">

**Author:** ![Michael\_Madden](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michael_madden/32/46640_2.png) [@Michael\_Madden](https://discuss.elastic.co/u/Michael_Madden)\
**Post date:** [June 24, 2019, 3:45pm UTC](https://discuss.elastic.co/t/ignore-lines-which-are-not-json-in-log-file-using-filebeat/187154/2 "2019-06-24T15:45:36Z")

</div>

The `include_lines` [option](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html#filebeat-input-log-include-lines) sounds like what you're looking for.

The following configuration would only include lines that started with a {. If the line does not look like JSON, it would not be exported.

```auto
filebeat.inputs:
- type: log
  ...
  include_lines: ['^{']

```

---

<div class="post-metadata">

**Author:** ![Jitendra\_Kumhar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jitendra_kumhar/32/48640_2.png) [@Jitendra\_Kumhar](https://discuss.elastic.co/u/Jitendra_Kumhar)\
**Post date:** [June 25, 2019, 6:33am UTC](https://discuss.elastic.co/t/ignore-lines-which-are-not-json-in-log-file-using-filebeat/187154/3 "2019-06-25T06:33:02Z")

</div>

Hey,  
Using this a log line which is not a valid json is still passed to elasticsearch. Can it be ignored?

---

<div class="post-metadata">

**Author:** ![sjabiulla](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sjabiulla/32/48429_2.png) [@sjabiulla](https://discuss.elastic.co/u/sjabiulla)\
**Post date:** [June 25, 2019, 7:48am UTC](https://discuss.elastic.co/t/ignore-lines-which-are-not-json-in-log-file-using-filebeat/187154/4 "2019-06-25T07:48:57Z")

</div>

If you are using Logstash, you can use [json](https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html) filter. It will parse only the json formatted events and rest will throw parse failure error and this error can be detected by `_jsonparsefailure` tag in the output.

Before indexing the output to ElasticSearch, you can have a conditional as below to ignore parse failures.

```
output{
     if "_jsonparsefailure" not in [tags] {
      elasticsearch{
				hosts => ["localhost:9200"]
				index => ["index_name-%{+YYYY.MM.dd}"]
			}
    }
}
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 23, 2019, 7:48am UTC](https://discuss.elastic.co/t/ignore-lines-which-are-not-json-in-log-file-using-filebeat/187154/5 "2019-07-23T07:48:58Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.