# How to filter json log file in logstash and send it to elastic search

**URL:** <https://discuss.elastic.co/t/how-to-filter-json-log-file-in-logstash-and-send-it-to-elastic-search/155981>\
**Category:** Logstash\
**Created:** [November 9, 2018, 6:00am UTC](https://discuss.elastic.co/t/how-to-filter-json-log-file-in-logstash-and-send-it-to-elastic-search/155981 "2018-11-09T06:00:50Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jin1129](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jin1129/32/37454_2.png) [@Jin1129](https://discuss.elastic.co/u/Jin1129)\
**Post date:** [November 9, 2018, 6:00am UTC](https://discuss.elastic.co/t/how-to-filter-json-log-file-in-logstash-and-send-it-to-elastic-search/155981/1 "2018-11-09T06:00:50Z")

</div>

This is my json log file. I'm trying to store the file to my elastic-Search through my logstash.

```auto
    { "id": "135569", "title" : "Star Trek Beyond", "year":2016 , "genre": 
    ["Action", "Adventure", "Sci-Fi"] }

```

after storing the data into the elasticSearch, my results is as follow

```auto
{
    "_index": "filebeat-6.2.4-2018.11.09",
    "_type": "doc",
    "_id": "n-J39mYB6zb53NvEugMO",
    "_score": 1,
    "_source": {
      "@timestamp": "2018-11-09T03:15:32.262Z",
      "source": "/Users/jinwoopark/Jin/json_files/testJson.log",
      "offset": 106,
      "message": """{ "id": "135569", "title" : "Star Trek Beyond", "year":2016 , "genre":["Action", "Adventure", "Sci-Fi"] }""",
      "id": "%{id}",
      "@version": "1",
      "host": "Jinui-MacBook-Pro.local",
      "tags": [
        "beats_input_codec_plain_applied"
      ],
      "prospector": {
        "type": "log"
      },
      "title": "%{title}",
      "beat": {
        "name": "Jinui-MacBook-Pro.local",
        "hostname": "Jinui-MacBook-Pro.local",
        "version": "6.2.4"
      }
    }
  }

```

What I'm trying to do is that,

I want to store only "genre value" into the message field, and store other values(ex id, title) into extra fields(the created fields, which is id and title field). but the extra fields were stored with empty values(%{id}, %{title}). It seems like I need to modify my logstash json filter, but here I need your help.

my current configuration of logstash is as follow

```auto
input {
    beats {
     port => 5044
    }
}

filter {
    json {
            source => "genre" //want to store only genre (from json log) into message field 
    }
    mutate {
            add_field => {
                    "id" => "%{[_source][message][id]}" // want to create extra field for id value from log file
                    "title" => "%{[_source][message][title]}" // want to create extra field for title value from log file
            }
    }
    date {
            match => ["timestamp", "dd/MM/yyyy:HH:mm:ss Z"]
    }
}
output {
    elasticsearch {
            hosts => ["http://localhost:9200"]
            index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    }
    stdout {
            codec => rubydebug
    }
}

```

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 9, 2018, 6:14am UTC](https://discuss.elastic.co/t/how-to-filter-json-log-file-in-logstash-and-send-it-to-elastic-search/155981/2 "2018-11-09T06:14:13Z")

</div>

You can [let Filebeat parse the JSON](https://www.elastic.co/guide/en/beats/filebeat/6.4/decode-json-fields.html) in the `message` field for you.

If you instead want to do it in Logstash you need to change your json filter to work off the `message` field (which contains the JSON data), and not the `genre` field which does not exist at this point.

You can then also remove the mutate filter and possibly also the date filter as I do not see any `timestamp` field in your data.

---

<div class="post-metadata">

**Author:** ![Jin1129](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jin1129/32/37454_2.png) [@Jin1129](https://discuss.elastic.co/u/Jin1129)\
**Post date:** [November 9, 2018, 7:34am UTC](https://discuss.elastic.co/t/how-to-filter-json-log-file-in-logstash-and-send-it-to-elastic-search/155981/3 "2018-11-09T07:34:14Z")

</div>

Thank you for your kind reply.

as you mentioned, I added extra fields in filebeat.yml as follow

```auto
processors:
 - decode_json_fields:
     fields: ["id", "title", "genre"]
     process_array: false
     max_depth: 3
     target: ""
     overwrite_keys: true

```

I can't find any difference. need extra help

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 9, 2018, 7:35am UTC](https://discuss.elastic.co/t/how-to-filter-json-log-file-in-logstash-and-send-it-to-elastic-search/155981/4 "2018-11-09T07:35:35Z")

</div>

The field you need to decode is the `message` field. This is the one containing the JSON data.

---

<div class="post-metadata">

**Author:** ![Jin1129](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jin1129/32/37454_2.png) [@Jin1129](https://discuss.elastic.co/u/Jin1129)\
**Post date:** [November 9, 2018, 7:44am UTC](https://discuss.elastic.co/t/how-to-filter-json-log-file-in-logstash-and-send-it-to-elastic-search/155981/5 "2018-11-09T07:44:50Z")

</div>

Thank you It's solved

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 7, 2018, 7:44am UTC](https://discuss.elastic.co/t/how-to-filter-json-log-file-in-logstash-and-send-it-to-elastic-search/155981/6 "2018-12-07T07:44:56Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.