# Filebeat is not forwarding log to logstach

**URL:** <https://discuss.elastic.co/t/filebeat-is-not-forwarding-log-to-logstach/159283>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [December 4, 2018, 4:34am UTC](https://discuss.elastic.co/t/filebeat-is-not-forwarding-log-to-logstach/159283 "2018-12-04T04:34:55Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Rakesh\_Thangapandian](https://avatars.discourse-cdn.com/v4/letter/r/c0e974/32.png) [@Rakesh\_Thangapandian](https://discuss.elastic.co/u/Rakesh_Thangapandian)\
**Post date:** [December 4, 2018, 4:34am UTC](https://discuss.elastic.co/t/filebeat-is-not-forwarding-log-to-logstach/159283/1 "2018-12-04T04:34:55Z")

</div>

Hi ,

I have created a dockerfile for filebeat and configured yaml files to listen for nginx module input and forward it to logstach. But the issue is my filebeat harvester didn't detect the nginx log update and so its not trying to forward.

\</\>filebeat dockerfile

```
FROM docker.elastic.co/beats/filebeat:6.5.1
USER root
COPY filebeat.yml /usr/share/filebeat/filebeat.yml 
RUN chmod go-w /usr/share/filebeat/filebeat.yml
USER filebeat

```

## ----End of dockerfile ------

## \</\> filebeat.yml file

```
filebeat.modules:
    module: nginx
    access:
      enabled: true
      var.paths: ["/var/log/nginx/access.log*"]
output.logstash:
   enabled: true
   hosts: ["hostid:5044"]

```

## Note: for logstash hostid will be replaced with actual ip address in my local and yml format and syntax is applied correctly. -----------End of filebeat.yml-----------------

## \</\> filebeat logs:

```
2018-12-04T04:15:40.378Z INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":260,"time":{"ms":9}},"total":{"ticks":460,"time":{"ms":9},"value":460},"user":{"ticks":200}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":5},"info":{"ephemeral_id":"57e2c532-fed1-460d-a05f-b33cf5be09d4","uptime":{"ms":360411}},"memstats":{"gc_next":4194304,"memory_alloc":2729352,"memory_total":7576760}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":2,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.03,"15":0.05,"5":0.09,"norm":{"1":0.0038,"15":0.0063,"5":0.0113}}}}}}
2018-12-04T04:16:10.382Z INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":290,"time":{"ms":22}},"total":{"ticks":490,"time":{"ms":22},"value":490},"user":{"ticks":200}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":5},"info":{"ephemeral_id":"57e2c532-fed1-460d-a05f-b33cf5be09d4","uptime":{"ms":390417}},"memstats":{"gc_next":4194304,"memory_alloc":1709176,"memory_total":7858320}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":2,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.02,"15":0.05,"5":0.08,"norm":{"1":0.0025,"15":0.0063,"5":0.01}}}}}}
2018-12-04T04:16:40.382Z INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":300,"time":{"ms":14}},"total":{"ticks":500,"time":{"ms":14},"value":500},"user":{"ticks":200}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":5},"info":{"ephemeral_id":"57e2c532-fed1-460d-a05f-b33cf5be09d4","uptime":{"ms":420412}},"memstats":{"gc_next":4194304,"memory_alloc":2001024,"memory_total":8150168}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":2,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.01,"15":0.05,"5":0.07,"norm":{"1":0.0013,"15":0.0063,"5":0.0088}}}}}}

```

## ------------ End of filebeat logs------------------------------

Any feedback to make it work could be helpful..

Thanks in advance`Preformatted text`

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [December 4, 2018, 2:13pm UTC](https://discuss.elastic.co/t/filebeat-is-not-forwarding-log-to-logstach/159283/2 "2018-12-04T14:13:49Z")

</div>

Pleae format logs, source code and config files using the `</>` button.

Do you have a more complete log file? It only logs metrics, but these state that you have no harvester running.

Can filebeat access the nginx log file? There is no harvester running, which might indicate that the file is not accessible from the filebeat container.

---

<div class="post-metadata">

**Author:** ![Rakesh\_Thangapandian](https://avatars.discourse-cdn.com/v4/letter/r/c0e974/32.png) [@Rakesh\_Thangapandian](https://discuss.elastic.co/u/Rakesh_Thangapandian)\
**Post date:** [December 5, 2018, 2:57am UTC](https://discuss.elastic.co/t/filebeat-is-not-forwarding-log-to-logstach/159283/3 "2018-12-05T02:57:34Z")

</div>

Hi steffens,

I have formatted the source and Log files.

**Do you have a more complete log file? It only logs metrics, but these state that you have no harvester running.**  
Yes, I have attached the detailed log in the Next reply

**Can filebeat access the nginx log file? There is no harvester running, which might indicate that the file is not accessible from the filebeat container.**  
I am not sure, and I just followed the steps from ([https://www.elastic.co/guide/en/beats/filebeat/current/running-on-docker.html](https://www.elastic.co/guide/en/beats/filebeat/current/running-on-docker.html)) and I gave USER root in the dockerfile which might gave the filebeat container a root permissions (If I am wrong please correct me and I am a beginner to docker and elastic, Kindly assist me to learn it).

---

<div class="post-metadata">

**Author:** ![Rakesh\_Thangapandian](https://avatars.discourse-cdn.com/v4/letter/r/c0e974/32.png) [@Rakesh\_Thangapandian](https://discuss.elastic.co/u/Rakesh_Thangapandian)\
**Post date:** [December 5, 2018, 2:58am UTC](https://discuss.elastic.co/t/filebeat-is-not-forwarding-log-to-logstach/159283/4 "2018-12-05T02:58:34Z")

</div>

Filebeat Complete Log

```
   WARNING: Error loading config file: /home/ubuntu/.docker/config.json: stat /home/ubuntu/.docker/config.json: permission denied
        2018-12-05T01:25:22.245Z INFO instance/beat.go:592 Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
        2018-12-05T01:25:22.421Z INFO instance/beat.go:599 Beat UUID: c51c266b-e551-4aab-a3e3-d283d24a1737
        2018-12-05T01:25:22.422Z INFO [seccomp] seccomp/seccomp.go:116 Syscall filter successfully installed
        2018-12-05T01:25:22.422Z INFO [beat] instance/beat.go:825 Beat info {"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "c51c266b-e551-4aab-a3e3-d283d24a1737"}}}
        2018-12-05T01:25:22.423Z INFO [beat] instance/beat.go:834 Build info {"system_info": {"build": {"commit": "b1c6ac83cc037bd75395334eed23b64bd63c87ef", "libbeat": "6.5.1", "time": "2018-11-16T01:29:47.000Z", "version": "6.5.1"}}}
        2018-12-05T01:25:22.423Z INFO [beat] instance/beat.go:837 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":8,"version":"go1.10.3"}}}
        2018-12-05T01:25:22.428Z INFO [beat] instance/beat.go:841 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2018-09-27T15:55:39Z","containerized":true,"name":"d21364c893db","ip":["127.0.0.1/8","172.17.0.2/16"],"kernel_version":"4.13.0-46-generic","mac":["02:42:ac:11:00:02"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":5,"patch":1804,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
        2018-12-05T01:25:22.430Z INFO [beat] instance/beat.go:870 Process info {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":null,"effective":null,"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2018-12-05T01:25:20.740Z"}}}
        2018-12-05T01:25:22.430Z INFO instance/beat.go:278 Setup Beat: filebeat; Version: 6.5.1
        2018-12-05T01:25:22.431Z INFO [publisher] pipeline/module.go:110 Beat name: d21364c893db
        2018-12-05T01:25:22.436Z INFO beater/filebeat.go:101 Enabled modules/filesets: nginx (access, error), ()
        2018-12-05T01:25:22.438Z INFO [monitoring] log/log.go:117 Starting metrics logging every 30s
        2018-12-05T01:25:22.439Z INFO instance/beat.go:400 filebeat start running.
        2018-12-05T01:25:22.439Z WARN beater/filebeat.go:180 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
        2018-12-05T01:25:22.439Z INFO registrar/registrar.go:97 No registry file found under: /usr/share/filebeat/data/registry. Creating a new registry file.
        2018-12-05T01:25:22.620Z INFO registrar/registrar.go:134 Loading registrar data from /usr/share/filebeat/data/registry
        2018-12-05T01:25:22.620Z INFO registrar/registrar.go:141 States Loaded from registrar: 0
        2018-12-05T01:25:22.620Z WARN beater/filebeat.go:374 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
        2018-12-05T01:25:22.620Z INFO crawler/crawler.go:72 Loading Inputs: 2
        2018-12-05T01:25:22.621Z INFO log/input.go:138 Configured paths: [/var/log/nginx/access.log*]
        2018-12-05T01:25:22.621Z INFO input/input.go:114 Starting input of type: log; ID: 12028807154522537067
        2018-12-05T01:25:22.622Z INFO log/input.go:138 Configured paths: [/var/log/nginx/error.log*]
        2018-12-05T01:25:22.622Z INFO input/input.go:114 Starting input of type: log; ID: 3451057047801109186
        2018-12-05T01:25:22.622Z INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 2
        2018-12-05T01:25:52.454Z INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":100,"time":{"ms":102}},"total":{"ticks":250,"time":{"ms":258},"value":250},"user":{"ticks":150,"time":{"ms":156}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":5},"info":{"ephemeral_id":"ca87e6a8-0791-494e-befc-31dd3bef8b45","uptime":{"ms":30233}},"memstats":{"gc_next":4194304,"memory_alloc":2826792,"memory_total":4412776,"rss":23511040}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"logstash"},"pipeline":{"clients":2,"events":{"active":0}}},"registrar":{"states":{"current":0},"writes":{"success":1,"total":1}},"system":{"cpu":{"cores":8},"load":{"1":0.14,"15":0.06,"5":0.13,"norm":{"1":0.0175,"15":0.0075,"5":0.0163}}}}}}
        2018-12-05T01:26:22.445Z INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":110,"time":{"ms":13}},"total":{"ticks":260,"time":{"ms":15},"value":260},"user":{"ticks":150,"time":{"ms":2}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":5},"info":{"ephemeral_id":"ca87e6a8-0791-494e-befc-31dd3bef8b45","uptime":{"ms":60220}},"memstats":{"gc_next":4194304,"memory_alloc":3204976,"memory_total":4790960}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":2,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.08,"15":0.06,"5":0.12,"norm":{"1":0.01,"15":0.0075,"5":0.015}}}}}}
```

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [December 5, 2018, 2:08pm UTC](https://discuss.elastic.co/t/filebeat-is-not-forwarding-log-to-logstach/159283/5 "2018-12-05T14:08:41Z")

</div>

Filebeat is not harvesting any logs: `"filebeat":{"harvester":{"open_files":0,"running":0}}`

Can you run `ls /var/log/nginx/*.log*` in your shell?

---

<div class="post-metadata">

**Author:** ![Rakesh\_Thangapandian](https://avatars.discourse-cdn.com/v4/letter/r/c0e974/32.png) [@Rakesh\_Thangapandian](https://discuss.elastic.co/u/Rakesh_Thangapandian)\
**Post date:** [December 5, 2018, 3:02pm UTC](https://discuss.elastic.co/t/filebeat-is-not-forwarding-log-to-logstach/159283/6 "2018-12-05T15:02:54Z")

</div>

\</\>This was the result of list cmd.

```
ubuntu@ubuntu-web:~$ ls /var/log/nginx/*.log*
    /var/log/nginx/access.log /var/log/nginx/access.log.14.gz /var/log/nginx/access.log.7.gz /var/log/nginx/error.log.11.gz /var/log/nginx/error.log.4.gz
    /var/log/nginx/access.log.1 /var/log/nginx/access.log.2.gz /var/log/nginx/access.log.8.gz /var/log/nginx/error.log.12.gz /var/log/nginx/error.log.5.gz
    /var/log/nginx/access.log.10.gz /var/log/nginx/access.log.3.gz /var/log/nginx/access.log.9.gz /var/log/nginx/error.log.13.gz /var/log/nginx/error.log.6.gz
    /var/log/nginx/access.log.11.gz /var/log/nginx/access.log.4.gz /var/log/nginx/error.log /var/log/nginx/error.log.14.gz /var/log/nginx/error.log.7.gz
    /var/log/nginx/access.log.12.gz /var/log/nginx/access.log.5.gz /var/log/nginx/error.log.1 /var/log/nginx/error.log.2.gz /var/log/nginx/error.log.8.gz
    /var/log/nginx/access.log.13.gz /var/log/nginx/access.log.6.gz /var/log/nginx/error.log.10.gz /var/log/nginx/error.log.3.gz /var/log/nginx/error.log.9.gz

```

**I guess mounting nginx log files directory inside filebeat container would solve the issue**

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [December 5, 2018, 3:47pm UTC](https://discuss.elastic.co/t/filebeat-is-not-forwarding-log-to-logstach/159283/7 "2018-12-05T15:47:15Z")

</div>

> I guess mounting nginx log files directory inside filebeat container would solve the issue

Yep, this would help. Docker containers isolate disks/volumes. Without explicitly mounting the right directories there is nothing for filebeat to read.

---

<div class="post-metadata">

**Author:** ![Rakesh\_Thangapandian](https://avatars.discourse-cdn.com/v4/letter/r/c0e974/32.png) [@Rakesh\_Thangapandian](https://discuss.elastic.co/u/Rakesh_Thangapandian)\
**Post date:** [December 6, 2018, 8:16am UTC](https://discuss.elastic.co/t/filebeat-is-not-forwarding-log-to-logstach/159283/8 "2018-12-06T08:16:22Z")

</div>

Yeah, Issue Resolved...  
Thank you So much @steffens

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 3, 2019, 8:16am UTC](https://discuss.elastic.co/t/filebeat-is-not-forwarding-log-to-logstach/159283/9 "2019-01-03T08:16:32Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.