# How to read json file using filebeat and send it to elasticsearch

**URL:** <https://discuss.elastic.co/t/how-to-read-json-file-using-filebeat-and-send-it-to-elasticsearch/91802>\
**Category:** Beats\
**Created:** [July 4, 2017, 5:13pm UTC](https://discuss.elastic.co/t/how-to-read-json-file-using-filebeat-and-send-it-to-elasticsearch/91802 "2017-07-04T17:13:48Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![aditya\_b](https://avatars.discourse-cdn.com/v4/letter/a/ed655f/32.png) [@aditya\_b](https://discuss.elastic.co/u/aditya_b)\
**Post date:** [July 4, 2017, 5:13pm UTC](https://discuss.elastic.co/t/how-to-read-json-file-using-filebeat-and-send-it-to-elasticsearch/91802/1 "2017-07-04T17:13:48Z")

</div>

How to read json file using filebeat and send it to elasticsearch.

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [July 4, 2017, 8:28pm UTC](https://discuss.elastic.co/t/how-to-read-json-file-using-filebeat-and-send-it-to-elasticsearch/91802/2 "2017-07-04T20:28:44Z")

</div>

You can follow the Filebeat [getting started guide](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-getting-started.html) to get Filebeat shipping the logs Elasticsearch. The only special thing you need to do is add the [json](https://www.elastic.co/guide/en/beats/filebeat/5.4/configuration-filebeat-options.html#config-json) configuration to the proscpector config so that Filebeat parses the JSON before sending it.

```auto
filebeat.prospectors:
- paths:
  - /var/log/mylog.json
  json.keys_under_root: true
  json.add_error_key: true

```

---

<div class="post-metadata">

**Author:** ![aditya\_b](https://avatars.discourse-cdn.com/v4/letter/a/ed655f/32.png) [@aditya\_b](https://discuss.elastic.co/u/aditya_b)\
**Post date:** [July 5, 2017, 6:12am UTC](https://discuss.elastic.co/t/how-to-read-json-file-using-filebeat-and-send-it-to-elasticsearch/91802/3 "2017-07-05T06:12:23Z")

</div>

I am able to send json file to elasticsearch and visualize in kibana. But i am not getting contents from json file.

After adding below lines, i am not able to start filebeat service.

- /var/log/mylog.json  
json.keys\_under\_root: true  
json.add\_error\_key: true

I want to parse the contents of json file and visualize the same in kibana.

Contents of Json:-

```auto
{"application":{"id":"d6a19b39-5e40-4cbb-9857-561df066be5b","securityResourceId":"a06e9992-be1f-4355-bb37-69331acf19be","name":"hello Application","description":"","created":1490177768938,"enforceCompleteSnapshots":false,"active":true,"tags":[],"deleted":false,"user":"admin"},"applicationProcess":{"id":"776c883b-7281-43ed-93eb-9dce669675a6","name":"hello App Process","description":"","active":true,"inventoryManagementType":"AUTOMATIC","offlineAgentHandling":"PRE_EXECUTION_CHECK","versionCount":2,"version":2,"commit":54,"path":"applications\/d6a19b39-5e40-4cbb-9857-561df066be5b\/processes\/776c883b-7281-43ed-93eb-9dce669675a6","deleted":false,"metadataType":"applicationProcess"},"environment":{"id":"6741e9d2-8e9a-4f80-b067-16eb96121149","securityResourceId":"13b02668-ad35-4031-aaf5-1dc82a4de72d","name":"helloDeploy","description":"","color":"#00B2EF","requireApprovals":false,"noSelfApprovals":false,"lockSnapshots":false,"calendarId":"d92cabe9-f96b-4637-9d62-d15bb26ec6c8","active":true,"deleted":false,"cleanupDaysToKeep":0,"cleanupCountToKeep":0,"enableProcessHistoryCleanup":false,"useSystemDefaultDays":true,"historyCleanupDaysToKeep":365,"conditions":[]},"id":"0ba25d74-8b9d-4fe9-8d80-f11feeec7ddf","submittedTime":1490246254582,"traceId":"bab5a5bc-4c87-4b9a-9481-fdffb9478384","userName":"admin","onlyChanged":true,"description":"","startTime":1490246255525,"result":"FAULTED","state":"CLOSED","paused":false,"endTime":1490246267119,"duration":11594},
{"application":{"id":"d6a19b39-5e40-4cbb-9857-561df066be5b","securityResourceId":"a06e9992-be1f-4355-bb37-69331acf19be","name":"hello Application","description":"","created":1490177768938,"enforceCompleteSnapshots":false,"active":true,"tags":[],"deleted":false,"user":"admin"},"applicationProcess":{"id":"776c883b-7281-43ed-93eb-9dce669675a6","name":"hello App Process","description":"","active":true,"inventoryManagementType":"AUTOMATIC","offlineAgentHandling":"PRE_EXECUTION_CHECK","versionCount":2,"version":2,"commit":54,"path":"applications\/d6a19b39-5e40-4cbb-9857-561df066be5b\/processes\/776c883b-7281-43ed-93eb-9dce669675a6","deleted":false,"metadataType":"applicationProcess"},"environment":{"id":"6741e9d2-8e9a-4f80-b067-16eb96121149","securityResourceId":"13b02668-ad35-4031-aaf5-1dc82a4de72d","name":"helloDeploy","description":"","color":"#00B2EF","requireApprovals":false,"noSelfApprovals":false,"lockSnapshots":false,"calendarId":"d92cabe9-f96b-4637-9d62-d15bb26ec6c8","active":true,"deleted":false,"cleanupDaysToKeep":0,"cleanupCountToKeep":0,"enableProcessHistoryCleanup":false,"useSystemDefaultDays":true,"historyCleanupDaysToKeep":365,"conditions":[]},"id":"97aa4b10-9422-42f7-849a-853c86e8a0d8","submittedTime":1490247166486,"traceId":"960cc7c8-fa7b-4077-b173-d3e97e632abe","userName":"admin","onlyChanged":true,"description":"","startTime":1490247167163,"result":"FAULTED","state":"CLOSED","paused":false,"endTime":1490247178590,"duration":11427},
{"application":{"id":"d6a19b39-5e40-4cbb-9857-561df066be5b","securityResourceId":"a06e9992-be1f-4355-bb37-69331acf19be","name":"hello Application","description":"","created":1490177768938,"enforceCompleteSnapshots":false,"active":true,"tags":[],"deleted":false,"user":"admin"},"applicationProcess":{"id":"776c883b-7281-43ed-93eb-9dce669675a6","name":"hello App Process","description":"","active":true,"inventoryManagementType":"AUTOMATIC","offlineAgentHandling":"PRE_EXECUTION_CHECK","versionCount":2,"version":2,"commit":54,"path":"applications\/d6a19b39-5e40-4cbb-9857-561df066be5b\/processes\/776c883b-7281-43ed-93eb-9dce669675a6","deleted":false,"metadataType":"applicationProcess"},"environment":{"id":"6741e9d2-8e9a-4f80-b067-16eb96121149","securityResourceId":"13b02668-ad35-4031-aaf5-1dc82a4de72d","name":"helloDeploy","description":"","color":"#00B2EF","requireApprovals":false,"noSelfApprovals":false,"lockSnapshots":false,"calendarId":"d92cabe9-f96b-4637-9d62-d15bb26ec6c8","active":true,"deleted":false,"cleanupDaysToKeep":0,"cleanupCountToKeep":0,"enableProcessHistoryCleanup":false,"useSystemDefaultDays":true,"historyCleanupDaysToKeep":365,"conditions":[]},"id":"b49ac86b-531a-48db-8cb5-2744a0a72126","submittedTime":1490247308405,"traceId":"6a9cb4b0-b385-4a14-b5f4-33778650bd3c","userName":"admin","onlyChanged":true,"description":"","startTime":1490247309105,"result":"SUCCEEDED","state":"CLOSED","paused":false,"endTime":1490247319644,"duration":10539},
{"application":{"id":"d6a19b39-5e40-4cbb-9857-561df066be5b","securityResourceId":"a06e9992-be1f-4355-bb37-69331acf19be","name":"hello Application","description":"","created":1490177768938,"enforceCompleteSnapshots":false,"active":true,"tags":[],"deleted":false,"user":"admin"},"applicationProcess":{"id":"776c883b-7281-43ed-93eb-9dce669675a6","name":"hello App Process","description":"","active":true,"inventoryManagementType":"AUTOMATIC","offlineAgentHandling":"PRE_EXECUTION_CHECK","versionCount":2,"version":2,"commit":54,"path":"applications\/d6a19b39-5e40-4cbb-9857-561df066be5b\/processes\/776c883b-7281-43ed-93eb-9dce669675a6","deleted":false,"metadataType":"applicationProcess"},"environment":{"id":"6741e9d2-8e9a-4f80-b067-16eb96121149","securityResourceId":"13b02668-ad35-4031-aaf5-1dc82a4de72d","name":"helloDeploy","description":"","color":"#00B2EF","requireApprovals":false,"noSelfApprovals":false,"lockSnapshots":false,"calendarId":"d92cabe9-f96b-4637-9d62-d15bb26ec6c8","active":true,"deleted":false,"cleanupDaysToKeep":0,"cleanupCountToKeep":0,"enableProcessHistoryCleanup":false,"useSystemDefaultDays":true,"historyCleanupDaysToKeep":365,"conditions":[]},"id":"915d0f0b-b7ac-4932-b541-2c07b1467997","submittedTime":1490247735457,"traceId":"fbe35578-5185-4370-b975-71e9f29ff45d","userName":"admin","onlyChanged":true,"description":"","startTime":1490247735871,"result":"SUCCEEDED","state":"CLOSED","paused":false,"endTime":1490247736008,"duration":137},

```

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [July 5, 2017, 2:39pm UTC](https://discuss.elastic.co/t/how-to-read-json-file-using-filebeat-and-send-it-to-elasticsearch/91802/4 "2017-07-05T14:39:12Z")

</div>

> [@aditya\_b](#):
>
> After adding below lines, i am not able to start filebeat service.

That sounds like a config issue. Please share your complete config file. And also the result of running `sudo filebeat.sh -e -configtest`.

---

<div class="post-metadata">

**Author:** ![aditya\_b](https://avatars.discourse-cdn.com/v4/letter/a/ed655f/32.png) [@aditya\_b](https://discuss.elastic.co/u/aditya_b)\
**Post date:** [July 5, 2017, 3:49pm UTC](https://discuss.elastic.co/t/how-to-read-json-file-using-filebeat-and-send-it-to-elasticsearch/91802/5 "2017-07-05T15:49:11Z")

</div>

Please find result of configtest as below :

```auto
2017/07/05 15:45:52.194188 beat.go:285: INFO Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2017/07/05 15:45:52.194211 beat.go:186: INFO Setup Beat: filebeat; Version: 5.4.3
2017/07/05 15:45:52.194291 logstash.go:90: INFO Max Retries set to: 3
2017/07/05 15:45:52.194344 outputs.go:108: INFO Activated logstash as output plugin.
2017/07/05 15:45:52.194383 metrics.go:23: INFO Metrics logging every 30s
2017/07/05 15:45:52.194399 publish.go:295: INFO Publisher name: ip-192-168-1-61.ec2.internal
2017/07/05 15:45:52.194536 async.go:63: INFO Flush Interval set to: 1s
2017/07/05 15:45:52.194547 async.go:64: INFO Max Bulk Size set to: 1024
Config OK

```

Also find my configuration file :

```auto
filebeat:
  prospectors:
    -
      paths:
        - /var/log/*.json
      # - /var/log/messages
      # - /var/log/*.log

      input_type: log

      document_type: syslog

  registry_file: /var/lib/filebeat/registry

output:
  logstash:
    hosts: ["192.168.1.61:5044"]
    bulk_max_size: 1024

    tls:
      certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]

shipper:

logging:
  files:
    rotateeverybytes: 10485760 # = 10MB

```

Please suggest.

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [July 5, 2017, 6:06pm UTC](https://discuss.elastic.co/t/how-to-read-json-file-using-filebeat-and-send-it-to-elasticsearch/91802/6 "2017-07-05T18:06:20Z")

</div>

Is that the config where the service starts normally? That config does not include the JSON config options.

When the Filebeat service fails to start, what's in the log file?

Did you migrate from Filebeat 1.x? The config looks like it's from 1.x. Some of the options have been renamed. Most importantly, `tls` was changed to `ssl` to be consistent with other Elastic projects. How about using this below as your filebeat.yml and see if it works.

```auto
filebeat.prospectors:
- paths:
  - /var/log/*.json
  document_type: syslog
  json.keys_under_root: true
  json.add_error_key: true

output.logstash:
  hosts: ["192.168.1.61:5044"]
  ssl.certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]

```

---

<div class="post-metadata">

**Author:** ![aditya\_b](https://avatars.discourse-cdn.com/v4/letter/a/ed655f/32.png) [@aditya\_b](https://discuss.elastic.co/u/aditya_b)\
**Post date:** [July 6, 2017, 3:57am UTC](https://discuss.elastic.co/t/how-to-read-json-file-using-filebeat-and-send-it-to-elasticsearch/91802/7 "2017-07-06T03:57:27Z")

</div>

Hi Andrew,

If i edit filebeat.yml file with the given configuration, it gives error when i start filebeat service. Here is status result after starting the filebeat service:  
● filebeat.service - filebeat  
Loaded: loaded (/usr/lib/systemd/system/filebeat.service; enabled; vendor preset: disabled)  
Active: failed (Result: start-limit) since Wed 2017-07-05 23:36:43 EDT; 11s ago  
Docs: [https://www.elastic.co/guide/en/beats/filebeat/current/index.html](https://www.elastic.co/guide/en/beats/filebeat/current/index.html)  
Process: 22754 ExecStart=/usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat (code=exited, status=1/FAILURE)  
Main PID: 22754 (code=exited, status=1/FAILURE)

Jul 05 23:36:43 ip-192-168-1-61.ec2.internal systemd[1]: filebeat.service: main process exited, code=exited, status=1/FAILURE  
Jul 05 23:36:43 ip-192-168-1-61.ec2.internal systemd[1]: Unit filebeat.service entered failed state.  
Jul 05 23:36:43 ip-192-168-1-61.ec2.internal systemd[1]: filebeat.service failed.  
Jul 05 23:36:43 ip-192-168-1-61.ec2.internal systemd[1]: filebeat.service holdoff time over, scheduling restart.  
Jul 05 23:36:43 ip-192-168-1-61.ec2.internal systemd[1]: start request repeated too quickly for filebeat.service  
Jul 05 23:36:43 ip-192-168-1-61.ec2.internal systemd[1]: Failed to start filebeat.  
Jul 05 23:36:43 ip-192-168-1-61.ec2.internal systemd[1]: Unit filebeat.service entered failed state.  
Jul 05 23:36:43 ip-192-168-1-61.ec2.internal systemd[1]:

Current version of filebeat is :-  
[root@ip-192-168-1-61 bin]# ./filebeat -version  
filebeat version 5.4.3 (amd64), libbeat 5.4.3

We can skip tls/ssl as of now because i have installed filebeat on same server on which ELK is setup.

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [July 6, 2017, 3:37pm UTC](https://discuss.elastic.co/t/how-to-read-json-file-using-filebeat-and-send-it-to-elasticsearch/91802/8 "2017-07-06T15:37:56Z")

</div>

> [@andrewkroh](#):
>
> When the Filebeat service fails to start, what's in the log file?

The log file should be located in `/var/log/filebeat/`, can you post what's in the file.

---

<div class="post-metadata">

**Author:** ![aditya\_b](https://avatars.discourse-cdn.com/v4/letter/a/ed655f/32.png) [@aditya\_b](https://discuss.elastic.co/u/aditya_b)\
**Post date:** [July 6, 2017, 5:36pm UTC](https://discuss.elastic.co/t/how-to-read-json-file-using-filebeat-and-send-it-to-elasticsearch/91802/9 "2017-07-06T17:36:30Z")

</div>

Hi,

Please find the attached file.

Thanks & Regards,  
Aditya Balhara

---

<div class="post-metadata">

**Author:** ![aditya\_b](https://avatars.discourse-cdn.com/v4/letter/a/ed655f/32.png) [@aditya\_b](https://discuss.elastic.co/u/aditya_b)\
**Post date:** [July 6, 2017, 5:41pm UTC](https://discuss.elastic.co/t/how-to-read-json-file-using-filebeat-and-send-it-to-elasticsearch/91802/10 "2017-07-06T17:41:09Z")

</div>

Filebeat last few lines :  
2017-07-06T12:54:01-04:00 INFO No non-zero metrics in the last 30s  
2017-07-06T12:54:31-04:00 INFO No non-zero metrics in the last 30s  
2017-07-06T12:54:47-04:00 INFO File is inactive: /var/log/Deployment\_summary.json. Closing because close\_inactive of 5m0s reached.  
2017-07-06T12:54:52-04:00 INFO Harvester started for file: /var/log/Deployment\_summary.json  
2017-07-06T12:55:01-04:00 INFO Non-zero metrics in the last 30s: filebeat.harvester.closed=1 filebeat.harvester.started=1 publish.events=2 registrar.states.update=2 registrar.writes=2  
2017-07-06T12:55:31-04:00 INFO No non-zero metrics in the last 30s  
2017-07-06T12:56:01-04:00 INFO No non-zero metrics in the last 30s  
2017-07-06T12:56:31-04:00 INFO No non-zero metrics in the last 30s  
2017-07-06T12:57:01-04:00 INFO No non-zero metrics in the last 30s  
2017-07-06T12:57:31-04:00 INFO No non-zero metrics in the last 30s  
2017-07-06T12:58:01-04:00 INFO No non-zero metrics in the last 30s  
2017-07-06T12:58:31-04:00 INFO No non-zero metrics in the last 30s  
2017-07-06T12:59:01-04:00 INFO No non-zero metrics in the last 30s  
2017-07-06T12:59:31-04:00 INFO No non-zero metrics in the last 30s  
2017-07-06T12:59:57-04:00 INFO File is inactive: /var/log/Deployment\_summary.json. Closing because close\_inactive of 5m0s reached.  
2017-07-06T13:00:01-04:00 INFO Non-zero metrics in the last 30s: filebeat.harvester.closed=1 filebeat.harvester.open\_files=-1 filebeat.harvester.running=-1 publish.events=1 registrar.states.update=1 registrar.writes=1  
2017-07-06T13:00:02-04:00 INFO Harvester started for file: /var/log/Deployment\_summary.json  
2017-07-06T13:00:31-04:00 INFO Non-zero metrics in the last 30s: filebeat.harvester.open\_files=1 filebeat.harvester.running=1 filebeat.harvester.started=1 publish.events=1 registrar.states.update=1 registrar.writes=1  
2017-07-06T13:01:01-04:00 INFO No non-zero metrics in the last 30s  
2017-07-06T13:01:31-04:00 INFO No non-zero metrics in the last 30s  
2017-07-06T13:02:01-04:00 INFO No non-zero metrics in the last 30s  
2017-07-06T13:02:31-04:00 INFO No non-zero metrics in the last 30s  
2017-07-06T13:03:01-04:00 INFO No non-zero metrics in the last 30s  
2017-07-06T13:03:31-04:00 INFO No non-zero metrics in the last 30s  
2017-07-06T13:04:01-04:00 INFO No non-zero metrics in the last 30s  
2017-07-06T13:04:31-04:00 INFO No non-zero metrics in the last 30s  
2017-07-06T13:05:01-04:00 INFO No non-zero metrics in the last 30s  
2017-07-06T13:05:07-04:00 INFO File is inactive: /var/log/Deployment\_summary.json. Closing because close\_inactive of 5m0s reached.  
2017-07-06T13:05:12-04:00 INFO Harvester started for file: /var/log/Deployment\_summary.json  
2017-07-06T13:05:31-04:00 INFO Non-zero metrics in the last 30s: filebeat.harvester.closed=1 filebeat.harvester.started=1 publish.events=2 registrar.states.update=2 registrar.writes=2  
2017-07-06T13:06:01-04:00 INFO No non-zero metrics in the last 30s  
2017-07-06T13:06:31-04:00 INFO No non-zero metrics in the last 30s  
2017-07-06T13:07:01-04:00 INFO No non-zero metrics in the last 30s  
2017-07-06T13:07:31-04:00 INFO No non-zero metrics in the last 30s  
2017-07-06T13:08:01-04:00 INFO No non-zero metrics in the last 30s  
2017-07-06T13:08:31-04:00 INFO No non-zero metrics in the last 30s  
2017-07-06T13:09:01-04:00 INFO No non-zero metrics in the last 30s  
2017-07-06T13:09:31-04:00 INFO No non-zero metrics in the last 30s  
2017-07-06T13:10:01-04:00 INFO No non-zero metrics in the last 30s  
2017-07-06T13:10:18-04:00 INFO File is inactive: /var/log/Deployment\_summary.json. Closing because close\_inactive of 5m0s reached.  
2017-07-06T13:10:23-04:00 INFO Harvester started for file: /var/log/Deployment\_summary.json  
2017-07-06T13:10:31-04:00 INFO Non-zero metrics in the last 30s: filebeat.harvester.closed=1 filebeat.harvester.started=1 publish.events=2 registrar.states.update=2 registrar.writes=2  
2017-07-06T13:11:01-04:00 INFO No non-zero metrics in the last 30s  
2017-07-06T13:11:31-04:00 INFO No non-zero metrics in the last 30s  
2017-07-06T13:12:01-04:00 INFO No non-zero metrics in the last 30s  
2017-07-06T13:12:31-04:00 INFO No non-zero metrics in the last 30s  
2017-07-06T13:13:01-04:00 INFO No non-zero metrics in the last 30s  
2017-07-06T13:13:31-04:00 INFO No non-zero metrics in the last 30s  
2017-07-06T13:14:01-04:00 INFO No non-zero metrics in the last 30s  
2017-07-06T13:14:31-04:00 INFO No non-zero metrics in the last 30s  
2017-07-06T13:15:01-04:00 INFO No non-zero metrics in the last 30s  
2017-07-06T13:15:28-04:00 INFO File is inactive: /var/log/Deployment\_summary.json. Closing because close\_inactive of 5m0s reached.  
2017-07-06T13:15:31-04:00 INFO Non-zero metrics in the last 30s: filebeat.harvester.closed=1 filebeat.harvester.open\_files=-1 filebeat.harvester.running=-1 publish.events=1 registrar.states.update=1 registrar.writes=1  
2017-07-06T13:15:33-04:00 INFO Harvester started for file: /var/log/Deployment\_summary.json  
2017-07-06T13:16:01-04:00 INFO Non-zero metrics in the last 30s: filebeat.harvester.open\_files=1 filebeat.harvester.running=1 filebeat.harvester.started=1 publish.events=1 registrar.states.update=1 registrar.writes=1  
2017-07-06T13:16:31-04:00 INFO No non-zero metrics in the last 30s  
2017-07-06T13:16:44-04:00 INFO Stopping filebeat  
2017-07-06T13:16:44-04:00 INFO Prospector channel stopped because beat is stopping.  
2017-07-06T13:16:44-04:00 INFO Stopping Crawler  
2017-07-06T13:16:44-04:00 INFO Stopping 1 prospectors  
2017-07-06T13:16:44-04:00 INFO Prospector ticker stopped  
2017-07-06T13:16:44-04:00 INFO Stopping Prospector: 806989121561951862  
2017-07-06T13:16:44-04:00 INFO Reader was closed: /var/log/Deployment\_summary.json. Closing.  
2017-07-06T13:16:44-04:00 INFO Crawler stopped  
2017-07-06T13:16:44-04:00 INFO Stopping spooler  
2017-07-06T13:16:44-04:00 INFO Stopping Registrar  
2017-07-06T13:16:44-04:00 INFO Ending Registrar  
2017-07-06T13:16:44-04:00 INFO Total non-zero values: filebeat.harvester.closed=142 filebeat.harvester.started=142 publish.events=287 registrar.states.current=11 registrar.states.update=287 registrar.writes=284  
2017-07-06T13:16:44-04:00 INFO Uptime: 12h9m42.415732288s  
2017-07-06T13:16:44-04:00 INFO filebeat stopped.

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [July 7, 2017, 2:30pm UTC](https://discuss.elastic.co/t/how-to-read-json-file-using-filebeat-and-send-it-to-elasticsearch/91802/11 "2017-07-07T14:30:26Z")

</div>

> [@aditya\_b](#):
>
> 2017-07-06T13:16:44-04:00 INFO Uptime: 12h9m42.415732288s  
> 2017-07-06T13:16:44-04:00 INFO filebeat stopped.

The log file indicates that Filebeat ran for 12 hours and stopped normally. That seems strange if the service is really not starting. I'm not sure what's going on. How about running through the commands below and sharing the output and maybe one of us can spot an issue (you might need to use [https://pastebin.com](https://pastebin.com) to share the output since it will be long).

```auto
uname -a
sudo filebeat.sh -e -d "*" -configtest
sudo cat /etc/filebeat/filebeat.yml
ps -ef | grep filebeat
sudo systemctl status filebeat.service
sudo systemctl stop filebeat.service
# Clear the registry so that it resends all log lines from the beginning.
sudo rm /var/lib/filebeat/registry
sudo filebeat.sh -e
# Let it run for a about a minute. Then Ctrl+C.
sudo systemctl start filebeat.service
sudo systemctl status filebeat.service
tail -100 /var/log/filebeat/filebeat

```

---

<div class="post-metadata">

**Author:** ![aditya\_b](https://avatars.discourse-cdn.com/v4/letter/a/ed655f/32.png) [@aditya\_b](https://discuss.elastic.co/u/aditya_b)\
**Post date:** [July 9, 2017, 10:34am UTC](https://discuss.elastic.co/t/how-to-read-json-file-using-filebeat-and-send-it-to-elasticsearch/91802/12 "2017-07-09T10:34:07Z")

</div>

Please find the configuration Test results :  
[root@ip-192-168-1-61 bin]# uname -a  
Linux ip-192-168-1-61.ec2.internal 3.10.0-514.21.2.el7.x86\_64 #1 SMP Sun May 28 17:08:21 EDT 2017 x86\_64 x86\_64 x86\_64 GNU/Linux  
[root@ip-192-168-1-61 bin]# sudo [filebeat.sh](http://filebeat.sh) -e -d "_" -configtest  
filebeat2017/07/09 10:26:04.378843 beat.go:339: CRIT Exiting: error loading config file: yaml: line 11: found a tab character that violate indentation  
Exiting: error loading config file: yaml: line 11: found a tab character that violate indentation  
[root@ip-192-168-1-61 bin]# sudo cat /etc/filebeat/filebeat.yml  
filebeat:  
prospectors:  
-  
paths:  
- /var/log/_.json  
# - /var/log/messages  
# - /var/log/\*.log

```
  input_type: log

  document_type: syslog
    json.keys_under_root: true
    json.add_error_key: true

```

registry\_file: /var/lib/filebeat/registry

output:  
logstash:  
hosts: ["192.168.1.61:5044"]  
bulk\_max\_size: 1024

```
tls:
  certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]

```

shipper:

logging:  
files:  
rotateeverybytes: 10485760 # = 10MB  
[root@ip-192-168-1-61 bin]# ps -ef | grep filebeat  
root 20029 18810 0 06:26 pts/0 00:00:00 grep --color=auto filebeat  
[root@ip-192-168-1-61 bin]# sudo systemctl status filebeat.service  
● filebeat.service - filebeat  
Loaded: loaded (/usr/lib/systemd/system/filebeat.service; enabled; vendor preset: disabled)  
Active: failed (Result: start-limit) since Sun 2017-07-09 06:25:47 EDT; 1min 2s ago  
Docs: [https://www.elastic.co/guide/en/beats/filebeat/current/index.html](https://www.elastic.co/guide/en/beats/filebeat/current/index.html)  
Process: 19867 ExecStart=/usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat (code=exited, status=1/FAILURE)  
Main PID: 19867 (code=exited, status=1/FAILURE)

Jul 09 06:25:46 ip-192-168-1-61.ec2.internal systemd[1]: filebeat.service: main process exited, code=exited, status=1/FAILURE  
Jul 09 06:25:46 ip-192-168-1-61.ec2.internal systemd[1]: Unit filebeat.service entered failed state.  
Jul 09 06:25:46 ip-192-168-1-61.ec2.internal systemd[1]: filebeat.service failed.  
Jul 09 06:25:47 ip-192-168-1-61.ec2.internal systemd[1]: filebeat.service holdoff time over, scheduling restart.  
Jul 09 06:25:47 ip-192-168-1-61.ec2.internal systemd[1]: start request repeated too quickly for filebeat.service  
Jul 09 06:25:47 ip-192-168-1-61.ec2.internal systemd[1]: Failed to start filebeat.  
Jul 09 06:25:47 ip-192-168-1-61.ec2.internal systemd[1]: Unit filebeat.service entered failed state.  
Jul 09 06:25:47 ip-192-168-1-61.ec2.internal systemd[1]: filebeat.service failed.  
[root@ip-192-168-1-61 bin]# sudo systemctl stop filebeat.service  
[root@ip-192-168-1-61 bin]# sudo rm /var/lib/filebeat/registry  
[root@ip-192-168-1-61 bin]# sudo [filebeat.sh](http://filebeat.sh) -e  
filebeat2017/07/09 10:27:27.196040 beat.go:339: CRIT Exiting: error loading config file: yaml: line 11: found a tab character that violate indentation  
Exiting: error loading config file: yaml: line 11: found a tab character that violate indentation  
[root@ip-192-168-1-61 bin]# sudo systemctl start filebeat.service  
[root@ip-192-168-1-61 bin]# sudo systemctl status filebeat.service  
● filebeat.service - filebeat  
Loaded: loaded (/usr/lib/systemd/system/filebeat.service; enabled; vendor preset: disabled)  
Active: failed (Result: start-limit) since Sun 2017-07-09 06:27:39 EDT; 4s ago  
Docs: [https://www.elastic.co/guide/en/beats/filebeat/current/index.html](https://www.elastic.co/guide/en/beats/filebeat/current/index.html)  
Process: 20220 ExecStart=/usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat (code=exited, status=1/FAILURE)  
Main PID: 20220 (code=exited, status=1/FAILURE)

Jul 09 06:27:38 ip-192-168-1-61.ec2.internal systemd[1]: filebeat.service: main process exited, code=exited, status=1/FAILURE  
Jul 09 06:27:38 ip-192-168-1-61.ec2.internal systemd[1]: Unit filebeat.service entered failed state.  
Jul 09 06:27:38 ip-192-168-1-61.ec2.internal systemd[1]: filebeat.service failed.  
Jul 09 06:27:39 ip-192-168-1-61.ec2.internal systemd[1]: filebeat.service holdoff time over, scheduling restart.  
Jul 09 06:27:39 ip-192-168-1-61.ec2.internal systemd[1]: start request repeated too quickly for filebeat.service  
Jul 09 06:27:39 ip-192-168-1-61.ec2.internal systemd[1]: Failed to start filebeat.  
Jul 09 06:27:39 ip-192-168-1-61.ec2.internal systemd[1]: Unit filebeat.service entered failed state.  
Jul 09 06:27:39 ip-192-168-1-61.ec2.internal systemd[1]: filebeat.service failed.  
[root@ip-192-168-1-61 bin]# tail -100 /var/log/filebeat/filebeat  
2017-07-09T06:22:58-04:00 INFO Setup Beat: filebeat; Version: 5.4.3  
2017-07-09T06:22:58-04:00 INFO Max Retries set to: 3  
2017-07-09T06:22:58-04:00 INFO Activated logstash as output plugin.  
2017-07-09T06:22:58-04:00 INFO Publisher name: ip-192-168-1-61.ec2.internal  
2017-07-09T06:22:58-04:00 INFO Flush Interval set to: 1s  
2017-07-09T06:22:58-04:00 INFO Max Bulk Size set to: 1024  
[root@ip-192-168-1-61 bin]#

---

<div class="post-metadata">

**Author:** ![aditya\_b](https://avatars.discourse-cdn.com/v4/letter/a/ed655f/32.png) [@aditya\_b](https://discuss.elastic.co/u/aditya_b)\
**Post date:** [July 9, 2017, 10:36am UTC](https://discuss.elastic.co/t/how-to-read-json-file-using-filebeat-and-send-it-to-elasticsearch/91802/13 "2017-07-09T10:36:48Z")

</div>

HI,

Also kept results over below link :  
[https://pastebin.com/mnQqpf03](https://pastebin.com/mnQqpf03)

Thanks in advance 🙂

---

<div class="post-metadata">

**Author:** ![aditya\_b](https://avatars.discourse-cdn.com/v4/letter/a/ed655f/32.png) [@aditya\_b](https://discuss.elastic.co/u/aditya_b)\
**Post date:** [July 13, 2017, 11:32am UTC](https://discuss.elastic.co/t/how-to-read-json-file-using-filebeat-and-send-it-to-elasticsearch/91802/14 "2017-07-13T11:32:05Z")

</div>

Hi Andrew,

Any update on same ?

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [July 14, 2017, 3:53am UTC](https://discuss.elastic.co/t/how-to-read-json-file-using-filebeat-and-send-it-to-elasticsearch/91802/15 "2017-07-14T03:53:29Z")

</div>

The logs you posted say

> Exiting: error loading config file: yaml: line 11: found a tab character that violate indentation

You need to remove all tab characters from the config file. YAML does not allow tabs. [YAML Ain't Markup Language](http://yaml.org/faq.html)

---

<div class="post-metadata">

**Author:** ![aditya\_b](https://avatars.discourse-cdn.com/v4/letter/a/ed655f/32.png) [@aditya\_b](https://discuss.elastic.co/u/aditya_b)\
**Post date:** [July 14, 2017, 4:53am UTC](https://discuss.elastic.co/t/how-to-read-json-file-using-filebeat-and-send-it-to-elasticsearch/91802/16 "2017-07-14T04:53:27Z")

</div>

When i start filebeat service without below json tags, filebeat service is working fine. ANd if i include the same in my filebeat.yml then filebeat service doesn't start.

```
json.keys_under_root: true
json.add_error_key: true

```

I am not using the tag now. But i have tried, tags is not the issue.

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [July 17, 2017, 4:38pm UTC](https://discuss.elastic.co/t/how-to-read-json-file-using-filebeat-and-send-it-to-elasticsearch/91802/17 "2017-07-17T16:38:42Z")

</div>

> [@aditya\_b](#):
>
> But i have tried, tags is not the issue.

The problem is tabs characters in your config file, not tags. See my previous post.

---

<div class="post-metadata">

**Author:** ![aditya\_b](https://avatars.discourse-cdn.com/v4/letter/a/ed655f/32.png) [@aditya\_b](https://discuss.elastic.co/u/aditya_b)\
**Post date:** [July 18, 2017, 3:59am UTC](https://discuss.elastic.co/t/how-to-read-json-file-using-filebeat-and-send-it-to-elasticsearch/91802/18 "2017-07-18T03:59:06Z")

</div>

Sorry it was written mistake, it is tab instead of tag.

filebeat service starts even when i am using the tabs but not including below codes.  
json.keys\_under\_root: true  
json.add\_error\_key: true

---

<div class="post-metadata">

**Author:** ![aditya\_b](https://avatars.discourse-cdn.com/v4/letter/a/ed655f/32.png) [@aditya\_b](https://discuss.elastic.co/u/aditya_b)\
**Post date:** [July 20, 2017, 10:01am UTC](https://discuss.elastic.co/t/how-to-read-json-file-using-filebeat-and-send-it-to-elasticsearch/91802/19 "2017-07-20T10:01:14Z")

</div>

Hi Andrew,

Any update on same ?

I have also removed tabs from the configuration file but not working. Please suggest asap.

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [July 20, 2017, 3:21pm UTC](https://discuss.elastic.co/t/how-to-read-json-file-using-filebeat-and-send-it-to-elasticsearch/91802/20 "2017-07-20T15:21:05Z")

</div>

If it is still now working, then what are the errors now? Please re-run these commands and share the output.

```auto
sudo filebeat.sh -e -d "*" -configtest
sudo cat /etc/filebeat/filebeat.yml
sudo filebeat.sh -e
# Let it run for a about a minute. Then Ctrl+C.
tail -100 /var/log/filebeat/filebeat

```

[Next page](https://discuss.elastic.co/t/how-to-read-json-file-using-filebeat-and-send-it-to-elasticsearch/91802.md?page=2)