# Using filebeat, cannot read json log to custom index

**URL:** <https://discuss.elastic.co/t/using-filebeat-cannot-read-json-log-to-custom-index/270914>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [April 22, 2021, 12:52am UTC](https://discuss.elastic.co/t/using-filebeat-cannot-read-json-log-to-custom-index/270914 "2021-04-22T00:52:09Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![thuynh](https://avatars.discourse-cdn.com/v4/letter/t/e274bd/32.png) [@thuynh](https://discuss.elastic.co/u/thuynh)\
**Post date:** [April 22, 2021, 12:52am UTC](https://discuss.elastic.co/t/using-filebeat-cannot-read-json-log-to-custom-index/270914/1 "2021-04-22T00:52:09Z")

</div>

Index template already created. It runs but doesn''t tell me any error and can't see new index which should be call "fb-message-7.12.0-yyyy.mm.dd"

I notice other issues i'm missing the param to indicate this is a custom json file trying to read, added the Processor part.

Can anybody help? still not telling me what is my issue, trying to load this custom json file to ES using custom index template

## test-es.json sample: {"communication\_medium\_id":1,"category\_name":"Appointment reminders","processed\_date":"2020/01/14 00:00:01","message\_fk":1182176}

Elasticsearch version 7.12.0

filebeat.yml below:

```auto
filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    #- /var/log/*.log
    - /home/test-es.json

# ======================= Elasticsearch template setting =======================
setup.template.overwrite: true
setup.ilm.enabled: false
setup.template.enabled: true
setup.template.name: "fb-message-%{[agent.version]}"
setup.template.pattern: "fb-message-%{[agent.version]}-*"

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["ip:9200"]
  index: "fb-message-%{[agent.version]}-%{+yyyy.MM.dd}"
  # Protocol - either `http` (default) or `https`.
  #protocol: "https"
  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: "xxx"
  password: "xxx"

# ================================= Processors =================================
processors:
  - decode_json_fields:
      fields: ["communication_medium_id","category_name","Appointment","processed_date","message_fk"]
      process_array: false
      max_depth: 1
      target: ""
      overwrite_keys: false
      add_error_key: true
```

---

<div class="post-metadata">

**Author:** ![Marius\_Iversen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_iversen/32/68988_2.png) [@Marius\_Iversen](https://discuss.elastic.co/u/Marius_Iversen)\
**Post date:** [April 24, 2021, 10:02am UTC](https://discuss.elastic.co/t/using-filebeat-cannot-read-json-log-to-custom-index/270914/2 "2021-04-24T10:02:15Z")

</div>

There 2 things that could/would be the issue here:

1. When data is being read by a filebeat log input, the logline is put in the field `message`, so try to first change your processor to this:

2. Is your json in your logfile, one json per line (ndjson), or is one JSON spread over multiple lines? If it is the latter, you need to add support for multiline like so:

```auto
    - type: log
      enabled: true
      paths:
        - /home/test-es.json
      multiline.pattern: '^{'
      multiline.negate: true
      multiline.match: after

```

---

<div class="post-metadata">

**Author:** ![thuynh](https://avatars.discourse-cdn.com/v4/letter/t/e274bd/32.png) [@thuynh](https://discuss.elastic.co/u/thuynh)\
**Post date:** [April 26, 2021, 9:47pm UTC](https://discuss.elastic.co/t/using-filebeat-cannot-read-json-log-to-custom-index/270914/3 "2021-04-26T21:47:34Z")

</div>

thanks for replying...actually my json suppose to be like this which works just testing ES in json file but even adding your suggestion doesn't work..

```auto
{"data": [{"communication_medium_id":1,"category_name":"Appointment reminders","processed_date":"2020/01/14 00:00:01","message_fk":1182176},{"communication_medium_id":221,"category_name":"Appointment reminders","processed_date":"2020/01/14 00:00:01","message_fk":1182176}]}

```

Not sure what i'm not getting but if I test with only one record json without the data: it works...but my file has many records..

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 24, 2021, 11:48pm UTC](https://discuss.elastic.co/t/using-filebeat-cannot-read-json-log-to-custom-index/270914/4 "2021-05-24T23:48:19Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.