# LogStash does not appear to be receiving files from FileBeat

**URL:** <https://discuss.elastic.co/t/logstash-does-not-appear-to-be-receiving-files-from-filebeat/68485>\
**Category:** Logstash\
**Created:** [December 8, 2016, 8:59pm UTC](https://discuss.elastic.co/t/logstash-does-not-appear-to-be-receiving-files-from-filebeat/68485 "2016-12-08T20:59:58Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![Airn5475](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/airn5475/32/13757_2.png) [@Airn5475](https://discuss.elastic.co/u/Airn5475)\
**Post date:** [December 8, 2016, 8:59pm UTC](https://discuss.elastic.co/t/logstash-does-not-appear-to-be-receiving-files-from-filebeat/68485/1 "2016-12-08T20:59:58Z")

</div>

I am new to Filebeats and LogStash and am feeling overwhelmed.  
My goal is to take .json files in through Filebeats to LogStash and finally onto Elasticsearch.  
I am currently working on my local dev machine running Windows 7.  
For right now I am just trying to take the files from Filebeats and output them to the console.

Here is my `filebeat.yml` file:

> filebeat.prospectors:
> 
> - input\_type: log  
> paths:
> - c:\Elasticsearch5\elasticsearch\data\*.json  
> output.logstash:  
> hosts: ["localhost:5043"]

As I put .json files into that I see them get picked up in my Filebeats console window when I run:

> C:\Program Files\Filebeat\>filebeat -e -c filebeat.yml -d "Publish"

Here is my `filebeat.conf` file that I use with LogStash:

> input {  
> beats {  
> port =\> 5043  
> codec =\> "json"  
> }  
> }  
> output {  
> stdout { codec =\> rubydebug }  
> }

Finally here is my `logstash.yml` file:

> node.name: logstash\_node\_aaron\_122016  
> config.reload.automatic: true  
> path.logs: c:\Elasticsearch5\log\logstash

I fire up LogStash using:

> C:\Elasticsearch5\logstash-5.0.2\>.\bin\logstash -f .\config\filebeat.conf --debug

I have been all over the internet looking at different documents and have had no success!  
Considering going back to using the ES API fed by a Console App!

Thank you!  
-Aaron

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [December 9, 2016, 6:36am UTC](https://discuss.elastic.co/t/logstash-does-not-appear-to-be-receiving-files-from-filebeat/68485/2 "2016-12-09T06:36:50Z")

</div>

If Filebeat doesn't send anything to Logstash there should be clues in the Filebeat log file.

---

<div class="post-metadata">

**Author:** ![Airn5475](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/airn5475/32/13757_2.png) [@Airn5475](https://discuss.elastic.co/u/Airn5475)\
**Post date:** [December 9, 2016, 2:33pm UTC](https://discuss.elastic.co/t/logstash-does-not-appear-to-be-receiving-files-from-filebeat/68485/3 "2016-12-09T14:33:44Z")

</div>

Thanks for the response. I am not getting any Log files in `C:\Program Files\Filebeat\logs` directory.  
I have the logging.level set to `debug` in my `filebeat.yml` file.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [December 9, 2016, 2:38pm UTC](https://discuss.elastic.co/t/logstash-does-not-appear-to-be-receiving-files-from-filebeat/68485/4 "2016-12-09T14:38:16Z")

</div>

What does the whole logging section look like? When posting make sure you preserve the indentation.

---

<div class="post-metadata">

**Author:** ![Airn5475](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/airn5475/32/13757_2.png) [@Airn5475](https://discuss.elastic.co/u/Airn5475)\
**Post date:** [December 9, 2016, 3:24pm UTC](https://discuss.elastic.co/t/logstash-does-not-appear-to-be-receiving-files-from-filebeat/68485/5 "2016-12-09T15:24:33Z")

</div>

```
#================================ Logging =====================================

# Sets log level. The default log level is info.
# Available log levels are: critical, error, warning, info, debug
logging.level: debug
# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
logging.selectors: ["*"]
```

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [December 9, 2016, 3:32pm UTC](https://discuss.elastic.co/t/logstash-does-not-appear-to-be-receiving-files-from-filebeat/68485/6 "2016-12-09T15:32:54Z")

</div>

Okay... so nothing about logs being sent to C:\Program Files\Filebeat\logs? Is that specified on the command line?

---

<div class="post-metadata">

**Author:** ![Airn5475](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/airn5475/32/13757_2.png) [@Airn5475](https://discuss.elastic.co/u/Airn5475)\
**Post date:** [December 9, 2016, 3:47pm UTC](https://discuss.elastic.co/t/logstash-does-not-appear-to-be-receiving-files-from-filebeat/68485/7 "2016-12-09T15:47:30Z")

</div>

No, I didn't specify it because it said in the documentation it would go to that folder by default:  
[https://www.elastic.co/guide/en/beats/filebeat/current/directory-layout.html](https://www.elastic.co/guide/en/beats/filebeat/current/directory-layout.html)

I updated the Logging section:  
#================================ Logging =====================================

```
# Sets log level. The default log level is info.
# Available log levels are: critical, error, warning, info, debug
logging.level: debug
# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
logging.selectors: ["*"]
logging.files:
  path: C:\Program Files\Filebeat\logs
  name: filebeat.log
  keepfiles: 7

```

I restarted Filebeat with these new settings and nothing appeared in the directory.

---

<div class="post-metadata">

**Author:** ![Airn5475](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/airn5475/32/13757_2.png) [@Airn5475](https://discuss.elastic.co/u/Airn5475)\
**Post date:** [December 9, 2016, 3:57pm UTC](https://discuss.elastic.co/t/logstash-does-not-appear-to-be-receiving-files-from-filebeat/68485/8 "2016-12-09T15:57:59Z")

</div>

I was going through this document and tried telnet on port 5043

> **[Common problems | Filebeat Reference \[8.11\] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/faq.html#connection-problem)**

> telnet localhost:5043  
> Connecting To localhost:5043...Could not open connection to the host, on port 23: Connect failed

Would this be part of the problem?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [December 9, 2016, 4:00pm UTC](https://discuss.elastic.co/t/logstash-does-not-appear-to-be-receiving-files-from-filebeat/68485/9 "2016-12-09T16:00:57Z")

</div>

That would certainly explain why Logstash isn't getting anything, but you really need to fix the logging too.

---

<div class="post-metadata">

**Author:** ![Airn5475](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/airn5475/32/13757_2.png) [@Airn5475](https://discuss.elastic.co/u/Airn5475)\
**Post date:** [December 9, 2016, 6:51pm UTC](https://discuss.elastic.co/t/logstash-does-not-appear-to-be-receiving-files-from-filebeat/68485/10 "2016-12-09T18:51:09Z")

</div>

I disabled Windows Firewall and nothing changed. No logs. Doesn't appear anything happening in LogStash.  
Here's the log from my Filebeat console window. Anything useful?

c:\Program Files\Filebeat\>filebeat -e -c filebeat.yml -d "\*"

logp.go:219: INFO Metrics logging every 30s  
beat.go:267: INFO Home path: [c:\Program Files\Filebeat] Config path: [c:\Program Files\Filebeat] Data path: [c:\Program Files\Filebeat\data] Logs path: [c:\Program Files\Filebeat\logs]  
beat.go:177: INFO Setup Beat: filebeat; Version: 5.1.1  
processor.go:43: DBG Processors:  
beat.go:183: DBG Initializing output plugins  
logstash.go:90: INFO Max Retries set to: 3  
outputs.go:106: INFO Activated logstash as output plugin.  
publish.go:234: DBG Create output worker  
publish.go:276: DBG No output is defined to store the topology. The server fields might not be filled.  
publish.go:291: INFO Publisher name: AARONB64WIN7  
async.go:63: INFO Flush Interval set to: 1s  
async.go:64: INFO Max Bulk Size set to: 2048  
async.go:72: DBG create bulk processing worker (interval=1s, bulk size=2048)  
beat.go:207: INFO filebeat start running.  
service\_windows.go:51: DBG Windows is interactive: true  
registrar.go:85: INFO Registry file set to: c:\Program Files\Filebeat\data\registry  
registrar.go:106: INFO Loading registrar data from c:\Program Files\Filebeat\data\registry  
registrar.go:131: INFO States Loaded from registrar: 3  
sync.go:41: INFO Start sending events to output  
registrar.go:230: INFO Starting Registrar  
crawler.go:34: INFO Loading Prospectors: 1  
spooler.go:63: INFO Starting spooler: spool\_size: 2048; idle\_timeout: 5s  
prospector\_log.go:41: DBG exclude\_files: []  
state.go:64: DBG New state added for c:\Elasticsearch5\elasticsearch\data\10.json  
state.go:64: DBG New state added for c:\Elasticsearch5\elasticsearch\data\20.json  
state.go:64: DBG New state added for c:\Elasticsearch5\elasticsearch\data\d2000.json  
prospector\_log.go:57: INFO Prospector with previous states loaded: 3  
prospector.go:69: DBG File Configs: [c:\Elasticsearch5\elasticsearch\data\*.json]  
crawler.go:46: INFO Loading Prospectors completed. Number of prospectors: 1  
crawler.go:61: INFO All prospectors are initialised and running with 3 states to persist  
crawler.go:56: DBG Starting prospector 0  
prospector.go:111: INFO Starting prospector of type: log  
prospector\_log.go:62: DBG Start next scan  
prospector\_log.go:212: DBG Check file for harvesting: c:\Elasticsearch5\elasticsearch\data\ABC.json  
prospector\_log.go:245: DBG Update existing file for harvesting: c:\Elasticsearch5\elasticsearch\data\ABC.json, offset: 0  
prospector\_log.go:254: DBG Resuming harvesting of file: c:\Elasticsearch5\elasticsearch\data\ABC.json, offset: 0  
log.go:256: DBG Setting offset for file based on seek: c:\Elasticsearch5\elasticsearch\data\ABC.json  
log.go:241: DBG Setting offset for file: c:\Elasticsearch5\elasticsearch\data\ABC.json. Offset: 0  
prospector\_log.go:212: DBG Check file for harvesting: c:\Elasticsearch5\elasticsearch\data\DEF.json  
log.go:84: INFO Harvester started for file: c:\Elasticsearch5\elasticsearch\data\ABC.json  
prospector\_log.go:245: DBG Update existing file for harvesting: c:\Elasticsearch5\elasticsearch\data\DEF.json, offset: 0  
log\_file.go:84: DBG End of file reached: c:\Elasticsearch5\elasticsearch\data\ABC.json; Backoff now.  
prospector\_log.go:254: DBG Resuming harvesting of file: c:\Elasticsearch5\elasticsearch\data\DEF.json, offset: 0  
log.go:256: DBG Setting offset for file based on seek: c:\Elasticsearch5\elasticsearch\data\DEF.json  
log.go:241: DBG Setting offset for file: c:\Elasticsearch5\elasticsearch\data\DEF.json. Offset: 0  
prospector\_log.go:83: DBG Prospector states cleaned up. Before: 3, After: 3  
log.go:84: INFO Harvester started for file: c:\Elasticsearch5\elasticsearch\data\DEF.json  
prospector\_log.go:99: DBG Remove state for file as file removed: c:\Elasticsearch5\elasticsearch\data\d2000.json  
log\_file.go:84: DBG End of file reached: c:\Elasticsearch5\elasticsearch\data\DEF.json; Backoff now.  
log\_file.go:84: DBG End of file reached: c:\Elasticsearch5\elasticsearch\data\ABC.json; Backoff now.  
log\_file.go:84: DBG End of file reached: c:\Elasticsearch5\elasticsearch\data\DEF.json; Backoff now.  
log\_file.go:84: DBG End of file reached: c:\Elasticsearch5\elasticsearch\data\ABC.json; Backoff now.  
log\_file.go:84: DBG End of file reached: c:\Elasticsearch5\elasticsearch\data\DEF.json; Backoff now.  
spooler.go:89: DBG Flushing spooler because of timeout. Events flushed: 6  
client.go:128: DBG No events to publish  
sync.go:68: DBG Events sent: 6  
registrar.go:269: DBG Processing 6 events  
state.go:109: DBG State removed for c:\Elasticsearch5\elasticsearch\data\d2000.json because of older: 0s  
registrar.go:255: DBG Registrar states cleaned up. Before: 3, After: 2  
registrar.go:292: DBG Write registry file: c:\Program Files\Filebeat\data\registry  
registrar.go:317: DBG Registry file updated. 2 states written.

---

<div class="post-metadata">

**Author:** ![Airn5475](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/airn5475/32/13757_2.png) [@Airn5475](https://discuss.elastic.co/u/Airn5475)\
**Post date:** [December 9, 2016, 7:11pm UTC](https://discuss.elastic.co/t/logstash-does-not-appear-to-be-receiving-files-from-filebeat/68485/11 "2016-12-09T19:11:45Z")

</div>

Okay I got the logs working - I had `-e` in the command and was disabling the file output;  
[https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-command-line.html](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-command-line.html)  
That's what I get for copying and pasting!

I ran the following and it output the log below:  
`c:\Program Files\Filebeat>filebeat -c filebeat.yml -d "*" -v -once`

DBG Disable stderr logging  
INFO Metrics logging every 30s  
INFO Home path: [c:\Program Files\Filebeat] Config path: [c:\Program Files\Filebeat] Data path: [c:\Program Files\Filebeat\data] Logs path: [c:\Program Files\Filebeat\logs]  
INFO Setup Beat: filebeat; Version: 5.1.1  
DBG Processors:  
DBG Initializing output plugins  
INFO Max Retries set to: 3  
INFO Activated logstash as output plugin.  
DBG Create output worker  
DBG No output is defined to store the topology. The server fields might not be filled.  
INFO Publisher name: AARONB64WIN7  
INFO Flush Interval set to: 1s  
INFO Max Bulk Size set to: 2048  
DBG create bulk processing worker (interval=1s, bulk size=2048)  
INFO filebeat start running.  
DBG Windows is interactive: true  
INFO Registry file set to: c:\Program Files\Filebeat\data\registry  
INFO Loading registrar data from c:\Program Files\Filebeat\data\registry  
INFO States Loaded from registrar: 3  
INFO Starting Registrar  
INFO Start sending events to output  
INFO Loading Prospectors: 1  
INFO Starting spooler: spool\_size: 2048; idle\_timeout: 5s  
DBG exclude\_files: []  
DBG New state added for c:\Elasticsearch5\elasticsearch\data\ABC1.json  
DBG New state added for c:\Elasticsearch5\elasticsearch\data\DEF2.json  
DBG New state added for c:\Elasticsearch5\elasticsearch\data\GHI3.json  
INFO Prospector with previous states loaded: 3  
DBG File Configs: [c:\Elasticsearch5\elasticsearch\data\*.json]  
INFO Loading Prospectors completed. Number of prospectors: 1  
INFO All prospectors are initialised and running with 3 states to persist  
DBG Starting prospector 0  
INFO Starting prospector of type: log  
INFO Running filebeat once. Waiting for completion ...  
DBG Start next scan  
DBG Check file for harvesting: c:\Elasticsearch5\elasticsearch\data\ABC11.json  
DBG Update existing file for harvesting: c:\Elasticsearch5\elasticsearch\data\ABC11.json, offset: 0  
DBG Resuming harvesting of file: c:\Elasticsearch5\elasticsearch\data\ABC11.json, offset: 0  
DBG Setting offset for file based on seek: c:\Elasticsearch5\elasticsearch\data\ABC11.json  
DBG Setting offset for file: c:\Elasticsearch5\elasticsearch\data\ABC11.json. Offset: 0  
DBG Check file for harvesting: c:\Elasticsearch5\elasticsearch\data\DEF22.json  
INFO Harvester started for file: c:\Elasticsearch5\elasticsearch\data\ABC11.json  
DBG Update existing file for harvesting: c:\Elasticsearch5\elasticsearch\data\DEF22.json, offset: 0  
DBG Resuming harvesting of file: c:\Elasticsearch5\elasticsearch\data\DEF22.json, offset: 0  
DBG Setting offset for file based on seek: c:\Elasticsearch5\elasticsearch\data\DEF22.json  
DBG End of file reached: c:\Elasticsearch5\elasticsearch\data\ABC11.json; Backoff now.  
DBG Setting offset for file: c:\Elasticsearch5\elasticsearch\data\DEF22.json. Offset: 0  
DBG Check file for harvesting: c:\Elasticsearch5\elasticsearch\data\GHI33.json  
INFO Harvester started for file: c:\Elasticsearch5\elasticsearch\data\DEF22.json  
DBG Update existing file for harvesting: c:\Elasticsearch5\elasticsearch\data\GHI33.json, offset: 0  
DBG Resuming harvesting of file: c:\Elasticsearch5\elasticsearch\data\GHI33.json, offset: 0  
DBG End of file reached: c:\Elasticsearch5\elasticsearch\data\DEF22.json; Backoff now.  
DBG Setting offset for file based on seek: c:\Elasticsearch5\elasticsearch\data\GHI33.json  
DBG Setting offset for file: c:\Elasticsearch5\elasticsearch\data\GHI33.json. Offset: 0  
DBG Prospector states cleaned up. Before: 3, After: 3  
INFO Harvester started for file: c:\Elasticsearch5\elasticsearch\data\GHI33.json  
DBG End of file reached: c:\Elasticsearch5\elasticsearch\data\GHI33.json; Backoff now.  
DBG End of file reached: c:\Elasticsearch5\elasticsearch\data\ABC11.json; Backoff now.  
DBG End of file reached: c:\Elasticsearch5\elasticsearch\data\DEF22.json; Backoff now.  
DBG End of file reached: c:\Elasticsearch5\elasticsearch\data\GHI33.json; Backoff now.  
DBG End of file reached: c:\Elasticsearch5\elasticsearch\data\ABC11.json; Backoff now.  
DBG End of file reached: c:\Elasticsearch5\elasticsearch\data\DEF22.json; Backoff now.  
DBG End of file reached: c:\Elasticsearch5\elasticsearch\data\GHI33.json; Backoff now.  
DBG Flushing spooler because of timeout. Events flushed: 6  
DBG No events to publish  
DBG Events sent: 6  
DBG Processing 6 events  
DBG Registrar states cleaned up. Before: 3, After: 3  
DBG Write registry file: c:\Program Files\Filebeat\data\registry  
DBG Registry file updated. 3 states written.  
DBG Received sigterm/sigint, stopping

---

<div class="post-metadata">

**Author:** ![Airn5475](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/airn5475/32/13757_2.png) [@Airn5475](https://discuss.elastic.co/u/Airn5475)\
**Post date:** [December 12, 2016, 3:10pm UTC](https://discuss.elastic.co/t/logstash-does-not-appear-to-be-receiving-files-from-filebeat/68485/12 "2016-12-12T15:10:52Z")

</div>

@magnusbaeck continuing my research...are there certain properties that have to be included in the .json file OR added to the "log" for LogStash to pick it up??  
I saw something about `@version` and `@timestamp`.

My json was as basic as possible:  
`{"ID":1, "Name":"Aaron"}`

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [December 13, 2016, 6:32am UTC](https://discuss.elastic.co/t/logstash-does-not-appear-to-be-receiving-files-from-filebeat/68485/13 "2016-12-13T06:32:43Z")

</div>

> are there certain properties that have to be included in the .json file OR added to the "log" for LogStash to pick it up??

No.

This seems more like a Filebeat problem now so I suggest moving this thread to that category or repost there to get the attention of the Beats folks.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 10, 2017, 6:33am UTC](https://discuss.elastic.co/t/logstash-does-not-appear-to-be-receiving-files-from-filebeat/68485/14 "2017-01-10T06:33:07Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.