# Filebeat 6.6 Hangs While Loading Log Data **URL:** \ **Category:** Elastic Training\ **Created:** [October 3, 2019, 11:29am UTC](https://discuss.elastic.co/t/filebeat-6-6-hangs-while-loading-log-data/202155 "2019-10-03T11:29:07Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![divdiff](https://avatars.discourse-cdn.com/v4/letter/d/2acd7d/32.png) [@divdiff](https://discuss.elastic.co/u/divdiff)\ **Post date:** [October 3, 2019, 11:29am UTC](https://discuss.elastic.co/t/filebeat-6-6-hangs-while-loading-log-data/202155/1 "2019-10-03T11:29:08Z")
I am prepping for the elasticsearch certification exam, and am trying to load the server logs data provided in ES Engineer 1 and 2 to my local cluster. I have three 6.6.1 nodes running on my laptop, with the following enabled in elasticsearch.yml: cluster.name: my\_cluster node.max\_local\_storage\_nodes: 3 xpack.security.enabled: true discovery.zen.minimum\_master\_nodes: 2 I'm using the following config for filebeat 6.6.1 to load the data to my cluster: filebeat.inputs: - type: log enabled: true paths: - /path/to/data/elastic\_blog\_curated\_access\_logs\_server1/\*.log processors: - decode\_json\_fields: fields: ['message'] target: '' overwrite\_keys: true - drop\_fields: fields: ["message", "prospector", "beat", "source", "offset"] setup.template.enabled: false output.elasticsearch: hosts: ["localhost:9200"] index: "logs\_server1" document\_type: "\_doc" bulk\_max\_size: 1000 user: "training" password: "heresmypassword" I run filebeat by executing ./filebeat -c filebeat.yml. Whenever I spin filebeat up, I receive the following in the logs: 2019-10-03T07:08:48.433-0400 INFO instance/beat.go:616 Home path: [/path/to/filebeat] Config path: [/path/to/filebeat] Data path: [/path/to/filebeat/data] Logs path: [/path/to/filebeat/logs] 2019-10-03T07:08:48.438-0400 INFO instance/beat.go:623 Beat UUID: 47b21207-302f-475b-94f1-3c15146e44b8 2019-10-03T07:08:48.438-0400 INFO [beat] instance/beat.go:936 Beat info {"system\_info": {"beat": {"path": {"config": "/path/to/filebeat", "data": "/path/to/filebeat/data", "home": "/path/to/filebeat", "logs": "/path/to/filebeat/logs"}, "type": "filebeat", "uuid": "47b21207-302f-475b-94f1-3c15146e44b8"}}} 2019-10-03T07:08:48.438-0400 INFO [beat] instance/beat.go:945 Build info {"system\_info": {"build": {"commit": "928f5e3f35fe28c1bd73513ff1cc89406eb212a6", "libbeat": "6.6.1", "time": "2019-02-13T16:12:11.000Z", "version": "6.6.1"}}} 2019-10-03T07:08:48.439-0400 INFO [beat] instance/beat.go:948 Go runtime info {"system\_info": {"go": {"os":"darwin","arch":"amd64","max\_procs":4,"version":"go1.10.8"}}} 2019-10-03T07:08:48.440-0400 INFO [beat] instance/beat.go:952 Host info {"system\_info": {"host": {"architecture":"x86\_64","boot\_time":"2019-09-12T05:50:47.425492-04:00","name":"my\_laptop\_name","ip":[.......],"kernel\_version":"18.7.0","mac":[.......],"os":{"family":"darwin","platform":"darwin","name":"Mac OS X","version":"10.14.6","major":10,"minor":14,"patch":6,"build":"18G95"},"timezone":"EDT","timezone\_offset\_sec":-14400,"id":"......."}}} 2019-10-03T07:08:48.441-0400 INFO [beat] instance/beat.go:981 Process info {"system\_info": {"process": {"cwd": "/path/to/filebeat", "exe": "./filebeat", "name": "filebeat", "pid": 32874, "ppid": 25428, "start\_time": "2019-10-03T07:08:48.307-0400"}}} 2019-10-03T07:08:48.441-0400 INFO instance/beat.go:281 Setup Beat: filebeat; Version: 6.6.1 2019-10-03T07:08:48.441-0400 INFO elasticsearch/client.go:165 Elasticsearch url: [http://localhost:9200](http://localhost:9200) 2019-10-03T07:08:48.444-0400 INFO [publisher] pipeline/module.go:110 Beat name: beat-name 2019-10-03T07:08:48.454-0400 INFO [monitoring] log/log.go:117 Starting metrics logging every 30s 2019-10-03T07:08:48.454-0400 INFO instance/beat.go:403 filebeat start running. 2019-10-03T07:08:48.454-0400 INFO registrar/registrar.go:134 Loading registrar data from /path/to/filebeat/data/registry 2019-10-03T07:08:48.462-0400 INFO registrar/registrar.go:141 States Loaded from registrar: 402 2019-10-03T07:08:48.462-0400 INFO crawler/crawler.go:72 Loading Inputs: 1 2019-10-03T07:08:48.644-0400 INFO log/input.go:138 Configured paths: [/path/to/data/elastic\_blog\_curated\_access\_logs\_server1/\*.log] 2019-10-03T07:08:48.644-0400 INFO input/input.go:114 Starting input of type: log; ID: 3080209040326827450 2019-10-03T07:08:48.644-0400 INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 1 2019-10-03T07:09:18.461-0400 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":98,"time":{"ms":98}},"total":{"ticks":260,"time":{"ms":260},"value":260},"user":{"ticks":162,"time":{"ms":162}}},"info":{"ephemeral\_id":"6af10b82-f593-4286-b461-86446b4c0b55","uptime":{"ms":30063}},"memstats":{"gc\_next":4194304,"memory\_alloc":2372312,"memory\_total":25198288,"rss":21200896}},"filebeat":{"events":{"added":134,"done":134},"harvester":{"open\_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"elasticsearch"},"pipeline":{"clients":1,"events":{"active":0,"filtered":134,"total":134}}},"registrar":{"states":{"current":402,"update":134},"writes":{"success":134,"total":134}},"system":{"cpu":{"cores":4},"load":{"1":1.6304,"15":2.3618,"5":2.2231,"norm":{"1":0.4076,"15":0.5905,"5":0.5558}}}}}} The last message continues to print, and filebeat hangs without loading any data into my cluster. Any ideas why this is happening? ---
**Author:** ![Melvyn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/melvyn/32/35269_2.png) [@Melvyn](https://discuss.elastic.co/u/Melvyn)\ **Post date:** [October 3, 2019, 11:34am UTC](https://discuss.elastic.co/t/filebeat-6-6-hangs-while-loading-log-data/202155/2 "2019-10-03T11:34:51Z")
I think the issue is probably related to the path of the files that you are trying to read. Can you confirm that you are using the absolute path to those files? ---
**Author:** ![divdiff](https://avatars.discourse-cdn.com/v4/letter/d/2acd7d/32.png) [@divdiff](https://discuss.elastic.co/u/divdiff)\ **Post date:** [October 3, 2019, 11:44am UTC](https://discuss.elastic.co/t/filebeat-6-6-hangs-while-loading-log-data/202155/3 "2019-10-03T11:44:11Z")
Yes I am. When populating the paths, I navigated to the respective directories and pulled the absolute path via pwd. ---
**Author:** ![dschneiter](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dschneiter/32/46162_2.png) [@dschneiter](https://discuss.elastic.co/u/dschneiter)\ **Post date:** [October 3, 2019, 11:49am UTC](https://discuss.elastic.co/t/filebeat-6-6-hangs-while-loading-log-data/202155/4 "2019-10-03T11:49:37Z")
So did you modify the path in your filebeats inputs configuration file that you posted abiove? (as it reads "/path/to/data", which is meant that you substitute the path to the absolute one matching your data folder) ---
**Author:** ![divdiff](https://avatars.discourse-cdn.com/v4/letter/d/2acd7d/32.png) [@divdiff](https://discuss.elastic.co/u/divdiff)\ **Post date:** [October 3, 2019, 11:51am UTC](https://discuss.elastic.co/t/filebeat-6-6-hangs-while-loading-log-data/202155/5 "2019-10-03T11:51:13Z")
That's right. I edited them from the actual paths I use. ---
**Author:** ![dschneiter](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dschneiter/32/46162_2.png) [@dschneiter](https://discuss.elastic.co/u/dschneiter)\ **Post date:** [October 3, 2019, 11:54am UTC](https://discuss.elastic.co/t/filebeat-6-6-hangs-while-loading-log-data/202155/6 "2019-10-03T11:54:09Z")
As you attended Engineer I and II can you share with us the version of the zip-file you are using can you test whether loading data via filebeat is working using the config files included in the zip file? I will try to assist you but as I have to teach the next couple of hours, I'm asking for a little bit of patience. ---
**Author:** ![Melvyn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/melvyn/32/35269_2.png) [@Melvyn](https://discuss.elastic.co/u/Melvyn)\ **Post date:** [October 3, 2019, 12:09pm UTC](https://discuss.elastic.co/t/filebeat-6-6-hangs-while-loading-log-data/202155/7 "2019-10-03T12:09:32Z")
Could you try to delete the registry file (it should be in the data folder) and restart Filebeat. Filebeat has probably already read your files and then is just waiting for new input. ---
**Author:** ![divdiff](https://avatars.discourse-cdn.com/v4/letter/d/2acd7d/32.png) [@divdiff](https://discuss.elastic.co/u/divdiff)\ **Post date:** [October 3, 2019, 12:53pm UTC](https://discuss.elastic.co/t/filebeat-6-6-hangs-while-loading-log-data/202155/8 "2019-10-03T12:53:01Z")
No worries Daniel. I'd like to sort this out by the end of the week, so I'm not in a rush. ---
**Author:** ![divdiff](https://avatars.discourse-cdn.com/v4/letter/d/2acd7d/32.png) [@divdiff](https://discuss.elastic.co/u/divdiff)\ **Post date:** [October 3, 2019, 12:54pm UTC](https://discuss.elastic.co/t/filebeat-6-6-hangs-while-loading-log-data/202155/9 "2019-10-03T12:54:33Z")
I think that did the trick. I'm seeing entries in the logs for filebeat detecting that there are files to consume. I'm getting a different error that looks like an authentication issue. I'll debug that separately. ---
**Author:** ![pmusa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pmusa/32/3498_2.png) [@pmusa](https://discuss.elastic.co/u/pmusa)\ **Post date:** [October 3, 2019, 11:56am UTC](https://discuss.elastic.co/t/filebeat-6-6-hangs-while-loading-log-data/202155/10 "2019-10-03T11:56:32Z")
The following needs to be updated to the correct path: "/path/to/data/elastic\_blog\_curated\_access\_logs\_server1/\*.log" Th "/path/to/filebeat" part needs to be updated with the path of your system. cheers ---
**Author:** ![divdiff](https://avatars.discourse-cdn.com/v4/letter/d/2acd7d/32.png) [@divdiff](https://discuss.elastic.co/u/divdiff)\ **Post date:** [October 3, 2019, 2:10pm UTC](https://discuss.elastic.co/t/filebeat-6-6-hangs-while-loading-log-data/202155/11 "2019-10-03T14:10:23Z")
I put that path there intentionally for this post. The path in filebeat.yml points to the correct directory. ---
**Author:** ![dschneiter](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dschneiter/32/46162_2.png) [@dschneiter](https://discuss.elastic.co/u/dschneiter)\ **Post date:** [October 4, 2019, 9:20am UTC](https://discuss.elastic.co/t/filebeat-6-6-hangs-while-loading-log-data/202155/12 "2019-10-04T09:20:46Z")
Hi Michael, am not sure, but is my understanding correct that the issue you faced is solved? Was it the registry file? Daniel ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 3, 2019, 9:20am UTC](https://discuss.elastic.co/t/filebeat-6-6-hangs-while-loading-log-data/202155/13 "2019-11-03T09:20:51Z")
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.