# Filebeat is not sending Json data to Elasticsearch

**URL:** <https://discuss.elastic.co/t/filebeat-is-not-sending-json-data-to-elasticsearch/103627>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [October 11, 2017, 10:20pm UTC](https://discuss.elastic.co/t/filebeat-is-not-sending-json-data-to-elasticsearch/103627 "2017-10-11T22:20:16Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![tatha](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tatha/32/22976_2.png) [@tatha](https://discuss.elastic.co/u/tatha)\
**Post date:** [October 11, 2017, 10:20pm UTC](https://discuss.elastic.co/t/filebeat-is-not-sending-json-data-to-elasticsearch/103627/1 "2017-10-11T22:20:16Z")

</div>

I am trying to send a set of json data (basically logs in the format of json) to elasticsearch instance using filebeat. Here are my config file

```
filebeat.prospectors:
- paths:
   - ~/PIE/Logs/*.json
  input_type: log
  json.keys_under_root: true
  json.add_error_key: true

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
  template.name: "filebeat"
  template.path: "filebeat.template.json" 

```

one of the the record within the file looks like below. All the records are in separate lines

`{"instance_id_str":"0","source_id_str":"APP/PROC/WEB","app_name_str":"App1","message":"hello","type":"syslog","event_uuid":"883d774f-02db-4967-9b89-eeff73890b09","origin_str":"rep","ALCH_TENANT_ID":"3213cd20-63cc-4592-b3ee-6a204769ce16","logmet_cluster":"topic3-elasticsearch_3","org_name_str":"Org1","@timestamp":"2017-09-29T02:40:50.519Z","message_type_str":"OUT","@version":"1","space_name_str":"prod","application_id_str":"3104b522-aba8-48e0-aef6-6291fc6f9250","ALCH_ACCOUNT_ID_str":"","org_id_str":"d728d5da-5346-4614-b092-e17be0f9b820","timestamp":"2017-09-29T02:40:50.519Z"}`

I ran the configtest

`sudo ./filebeat -configtest -e`

The output looks ok

```
2017/10/11 22:12:13.895468 beat.go:297: INFO Home path: [/Users/taroy/Software/filebeat-5.6.3-darwin-x86_64] Config path: [/Users/taroy/Software/filebeat-5.6.3-darwin-x86_64] Data path: [/Users/taroy/Software/filebeat-5.6.3-darwin-x86_64/data] Logs path: [/Users/taroy/Software/filebeat-5.6.3-darwin-x86_64/logs]
2017/10/11 22:12:13.895783 beat.go:192: INFO Setup Beat: filebeat; Version: 5.6.3
2017/10/11 22:12:13.895821 metrics.go:23: INFO Metrics logging every 30s
2017/10/11 22:12:13.895797 processor.go:44: DBG Processors: 
2017/10/11 22:12:13.896051 beat.go:198: DBG Initializing output plugins
2017/10/11 22:12:13.896508 output.go:258: INFO Loading template enabled. Reading template file: /Users/taroy/Software/filebeat-5.6.3-darwin-x86_64/filebeat.template.json
2017/10/11 22:12:13.898453 output.go:269: INFO Loading template enabled for Elasticsearch 2.x. Reading template file: /Users/taroy/Software/filebeat-5.6.3-darwin-x86_64/filebeat.template-es2x.json
2017/10/11 22:12:13.899530 output.go:281: INFO Loading template enabled for Elasticsearch 6.x. Reading template file: /Users/taroy/Software/filebeat-5.6.3-darwin-x86_64/filebeat.template-es6x.json
2017/10/11 22:12:13.900029 client.go:128: INFO Elasticsearch url: http://localhost:9200
2017/10/11 22:12:13.900388 outputs.go:108: INFO Activated elasticsearch as output plugin.
2017/10/11 22:12:13.900400 publish.go:243: DBG Create output worker
2017/10/11 22:12:13.900447 publish.go:285: DBG No output is defined to store the topology. The server fields might not be filled.
2017/10/11 22:12:13.900463 publish.go:300: INFO Publisher name: Tathas-Mac.local
2017/10/11 22:12:13.900533 async.go:63: INFO Flush Interval set to: 1s
2017/10/11 22:12:13.900540 async.go:64: INFO Max Bulk Size set to: 50
2017/10/11 22:12:13.900546 async.go:72: DBG create bulk processing worker (interval=1s, bulk size=50)
Config OK

```

Yet when i run the service

`sudo ./filebeat -e -c filebeat.yml -d "publish"`

The filebeat index is not created in my local elasticsearch instance and also the files under ~/PIE/Logs/ are not processed

Can someone please guide me in figuring out the problem

Thanks,  
Tatha

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [October 12, 2017, 11:12am UTC](https://discuss.elastic.co/t/filebeat-is-not-sending-json-data-to-elasticsearch/103627/2 "2017-10-12T11:12:36Z")

</div>

Do you have some filebeat log output?

---

<div class="post-metadata">

**Author:** ![tatha](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tatha/32/22976_2.png) [@tatha](https://discuss.elastic.co/u/tatha)\
**Post date:** [October 12, 2017, 2:12pm UTC](https://discuss.elastic.co/t/filebeat-is-not-sending-json-data-to-elasticsearch/103627/3 "2017-10-12T14:12:37Z")

</div>

Thanks Steffens for looking into it. Here is the log output

I have installed all the 3 products (ELstic, KIbana and FileBeats) and not using the docker containers. There is also no security in place.

```
Tathas-Mac:filebeat-5.6.3-darwin-x86_64 taroy$ sudo ./filebeat -e -c filebeat.yml -d "publish"
2017/10/11 22:29:13.029341 beat.go:297: INFO Home path: [/Users/taroy/Software/filebeat-5.6.3-darwin-x86_64] Config path: [/Users/taroy/Software/filebeat-5.6.3-darwin-x86_64] Data path: [/Users/taroy/Software/filebeat-5.6.3-darwin-x86_64/data] Logs path: [/Users/taroy/Software/filebeat-5.6.3-darwin-x86_64/logs]
2017/10/11 22:29:13.029418 metrics.go:23: INFO Metrics logging every 30s
2017/10/11 22:29:13.030011 beat.go:192: INFO Setup Beat: filebeat; Version: 5.6.3
2017/10/11 22:29:13.030470 output.go:258: INFO Loading template enabled. Reading template file: /Users/taroy/Software/filebeat-5.6.3-darwin-x86_64/filebeat.template.json
2017/10/11 22:29:13.033106 output.go:269: INFO Loading template enabled for Elasticsearch 2.x. Reading template file: /Users/taroy/Software/filebeat-5.6.3-darwin-x86_64/filebeat.template-es2x.json
2017/10/11 22:29:13.033661 output.go:281: INFO Loading template enabled for Elasticsearch 6.x. Reading template file: /Users/taroy/Software/filebeat-5.6.3-darwin-x86_64/filebeat.template-es6x.json
2017/10/11 22:29:13.035191 client.go:128: INFO Elasticsearch url: http://localhost:9200
2017/10/11 22:29:13.035229 outputs.go:108: INFO Activated elasticsearch as output plugin.
2017/10/11 22:29:13.035237 publish.go:243: DBG Create output worker
2017/10/11 22:29:13.035311 publish.go:285: DBG No output is defined to store the topology. The server fields might not be filled.
2017/10/11 22:29:13.035329 publish.go:300: INFO Publisher name: Tathas-Mac.local
2017/10/11 22:29:13.035404 async.go:63: INFO Flush Interval set to: 1s
2017/10/11 22:29:13.035412 async.go:64: INFO Max Bulk Size set to: 50
2017/10/11 22:29:13.035419 async.go:72: DBG create bulk processing worker (interval=1s, bulk size=50)
2017/10/11 22:29:13.036425 beat.go:233: INFO filebeat start running.
2017/10/11 22:29:13.036454 registrar.go:85: INFO Registry file set to: /Users/taroy/Software/filebeat-5.6.3-darwin-x86_64/data/registry
2017/10/11 22:29:13.036489 registrar.go:106: INFO Loading registrar data from /Users/taroy/Software/filebeat-5.6.3-darwin-x86_64/data/registry
2017/10/11 22:29:13.036515 registrar.go:123: INFO States Loaded from registrar: 0
2017/10/11 22:29:13.036546 crawler.go:38: INFO Loading Prospectors: 1
2017/10/11 22:29:13.036655 spooler.go:63: INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2017/10/11 22:29:13.037023 registrar.go:236: INFO Starting Registrar
2017/10/11 22:29:13.036655 prospector_log.go:65: INFO Prospector with previous states loaded: 0
2017/10/11 22:29:13.036595 sync.go:41: INFO Start sending events to output
2017/10/11 22:29:13.037370 prospector.go:124: INFO Starting prospector of type: log; id: 11576210144221994408 
2017/10/11 22:29:13.037385 crawler.go:58: INFO Loading and starting Prospectors completed. Enabled prospectors: 1
2017/10/11 22:29:43.031317 metrics.go:34: INFO No non-zero metrics in the last 30s
2017/10/11 22:30:13.030285 metrics.go:34: INFO No non-zero metrics in the last 30s
```

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [October 12, 2017, 4:03pm UTC](https://discuss.elastic.co/t/filebeat-is-not-sending-json-data-to-elasticsearch/103627/4 "2017-10-12T16:03:02Z")

</div>

No file is really picked up here. Is the file actually listed in the registry file? Does it work if you use the absolute paths to the log files (replace `~` with the actual path).

---

<div class="post-metadata">

**Author:** ![tatha](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tatha/32/22976_2.png) [@tatha](https://discuss.elastic.co/u/tatha)\
**Post date:** [October 12, 2017, 4:33pm UTC](https://discuss.elastic.co/t/filebeat-is-not-sending-json-data-to-elasticsearch/103627/5 "2017-10-12T16:33:46Z")

</div>

Thanks for the input. Yes that was the problem

Once I gave the absolute path /Users/taroy/.... it was able to pick the file.

Apologize for wasting your time on such a basic problem

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 9, 2017, 4:33pm UTC](https://discuss.elastic.co/t/filebeat-is-not-sending-json-data-to-elasticsearch/103627/6 "2017-11-09T16:33:56Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.