Filebeat is not forwarding log to logstach

Rakesh_Thangapandian · December 4, 2018, 4:34am

Hi ,

I have created a dockerfile for filebeat and configured yaml files to listen for nginx module input and forward it to logstach. But the issue is my filebeat harvester didn't detect the nginx log update and so its not trying to forward.

</>filebeat dockerfile

FROM docker.elastic.co/beats/filebeat:6.5.1
USER root
COPY filebeat.yml /usr/share/filebeat/filebeat.yml 
RUN chmod go-w /usr/share/filebeat/filebeat.yml
USER filebeat

----End of dockerfile ------

</> filebeat.yml file

filebeat.modules:
    module: nginx
    access:
      enabled: true
      var.paths: ["/var/log/nginx/access.log*"]
output.logstash:
   enabled: true
   hosts: ["hostid:5044"]

Note: for logstash hostid will be replaced with actual ip address in my local and yml format and syntax is applied correctly.
-----------End of filebeat.yml-----------------

</> filebeat logs:

2018-12-04T04:15:40.378Z        INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":260,"time":{"ms":9}},"total":{"ticks":460,"time":{"ms":9},"value":460},"user":{"ticks":200}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":5},"info":{"ephemeral_id":"57e2c532-fed1-460d-a05f-b33cf5be09d4","uptime":{"ms":360411}},"memstats":{"gc_next":4194304,"memory_alloc":2729352,"memory_total":7576760}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":2,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.03,"15":0.05,"5":0.09,"norm":{"1":0.0038,"15":0.0063,"5":0.0113}}}}}}
2018-12-04T04:16:10.382Z        INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":290,"time":{"ms":22}},"total":{"ticks":490,"time":{"ms":22},"value":490},"user":{"ticks":200}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":5},"info":{"ephemeral_id":"57e2c532-fed1-460d-a05f-b33cf5be09d4","uptime":{"ms":390417}},"memstats":{"gc_next":4194304,"memory_alloc":1709176,"memory_total":7858320}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":2,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.02,"15":0.05,"5":0.08,"norm":{"1":0.0025,"15":0.0063,"5":0.01}}}}}}
2018-12-04T04:16:40.382Z        INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":300,"time":{"ms":14}},"total":{"ticks":500,"time":{"ms":14},"value":500},"user":{"ticks":200}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":5},"info":{"ephemeral_id":"57e2c532-fed1-460d-a05f-b33cf5be09d4","uptime":{"ms":420412}},"memstats":{"gc_next":4194304,"memory_alloc":2001024,"memory_total":8150168}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":2,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.01,"15":0.05,"5":0.07,"norm":{"1":0.0013,"15":0.0063,"5":0.0088}}}}}}

------------ End of filebeat logs------------------------------

Any feedback to make it work could be helpful..

Thanks in advancePreformatted text

steffens · December 4, 2018, 2:13pm

Pleae format logs, source code and config files using the </> button.

Do you have a more complete log file? It only logs metrics, but these state that you have no harvester running.

Can filebeat access the nginx log file? There is no harvester running, which might indicate that the file is not accessible from the filebeat container.

Rakesh_Thangapandian · December 5, 2018, 2:57am

Hi steffens,

I have formatted the source and Log files.

Do you have a more complete log file? It only logs metrics, but these state that you have no harvester running.
Yes, I have attached the detailed log in the Next reply

Can filebeat access the nginx log file? There is no harvester running, which might indicate that the file is not accessible from the filebeat container.
I am not sure, and I just followed the steps from (https://www.elastic.co/guide/en/beats/filebeat/current/running-on-docker.html) and I gave USER root in the dockerfile which might gave the filebeat container a root permissions (If I am wrong please correct me and I am a beginner to docker and elastic, Kindly assist me to learn it).

Rakesh_Thangapandian · December 5, 2018, 2:58am

Filebeat Complete Log

   WARNING: Error loading config file: /home/ubuntu/.docker/config.json: stat /home/ubuntu/.docker/config.json: permission denied
        2018-12-05T01:25:22.245Z        INFO    instance/beat.go:592    Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
        2018-12-05T01:25:22.421Z        INFO    instance/beat.go:599    Beat UUID: c51c266b-e551-4aab-a3e3-d283d24a1737
        2018-12-05T01:25:22.422Z        INFO    [seccomp]       seccomp/seccomp.go:116  Syscall filter successfully installed
        2018-12-05T01:25:22.422Z        INFO    [beat]  instance/beat.go:825    Beat info       {"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "c51c266b-e551-4aab-a3e3-d283d24a1737"}}}
        2018-12-05T01:25:22.423Z        INFO    [beat]  instance/beat.go:834    Build info      {"system_info": {"build": {"commit": "b1c6ac83cc037bd75395334eed23b64bd63c87ef", "libbeat": "6.5.1", "time": "2018-11-16T01:29:47.000Z", "version": "6.5.1"}}}
        2018-12-05T01:25:22.423Z        INFO    [beat]  instance/beat.go:837    Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":8,"version":"go1.10.3"}}}
        2018-12-05T01:25:22.428Z        INFO    [beat]  instance/beat.go:841    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2018-09-27T15:55:39Z","containerized":true,"name":"d21364c893db","ip":["127.0.0.1/8","172.17.0.2/16"],"kernel_version":"4.13.0-46-generic","mac":["02:42:ac:11:00:02"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":5,"patch":1804,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
        2018-12-05T01:25:22.430Z        INFO    [beat]  instance/beat.go:870    Process info    {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":null,"effective":null,"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2018-12-05T01:25:20.740Z"}}}
        2018-12-05T01:25:22.430Z        INFO    instance/beat.go:278    Setup Beat: filebeat; Version: 6.5.1
        2018-12-05T01:25:22.431Z        INFO    [publisher]     pipeline/module.go:110  Beat name: d21364c893db
        2018-12-05T01:25:22.436Z        INFO    beater/filebeat.go:101  Enabled modules/filesets: nginx (access, error),  ()
        2018-12-05T01:25:22.438Z        INFO    [monitoring]    log/log.go:117  Starting metrics logging every 30s
        2018-12-05T01:25:22.439Z        INFO    instance/beat.go:400    filebeat start running.
        2018-12-05T01:25:22.439Z        WARN    beater/filebeat.go:180  Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
        2018-12-05T01:25:22.439Z        INFO    registrar/registrar.go:97       No registry file found under: /usr/share/filebeat/data/registry. Creating a new registry file.
        2018-12-05T01:25:22.620Z        INFO    registrar/registrar.go:134      Loading registrar data from /usr/share/filebeat/data/registry
        2018-12-05T01:25:22.620Z        INFO    registrar/registrar.go:141      States Loaded from registrar: 0
        2018-12-05T01:25:22.620Z        WARN    beater/filebeat.go:374  Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
        2018-12-05T01:25:22.620Z        INFO    crawler/crawler.go:72   Loading Inputs: 2
        2018-12-05T01:25:22.621Z        INFO    log/input.go:138        Configured paths: [/var/log/nginx/access.log*]
        2018-12-05T01:25:22.621Z        INFO    input/input.go:114      Starting input of type: log; ID: 12028807154522537067
        2018-12-05T01:25:22.622Z        INFO    log/input.go:138        Configured paths: [/var/log/nginx/error.log*]
        2018-12-05T01:25:22.622Z        INFO    input/input.go:114      Starting input of type: log; ID: 3451057047801109186
        2018-12-05T01:25:22.622Z        INFO    crawler/crawler.go:106  Loading and starting Inputs completed. Enabled inputs: 2
        2018-12-05T01:25:52.454Z        INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":100,"time":{"ms":102}},"total":{"ticks":250,"time":{"ms":258},"value":250},"user":{"ticks":150,"time":{"ms":156}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":5},"info":{"ephemeral_id":"ca87e6a8-0791-494e-befc-31dd3bef8b45","uptime":{"ms":30233}},"memstats":{"gc_next":4194304,"memory_alloc":2826792,"memory_total":4412776,"rss":23511040}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"logstash"},"pipeline":{"clients":2,"events":{"active":0}}},"registrar":{"states":{"current":0},"writes":{"success":1,"total":1}},"system":{"cpu":{"cores":8},"load":{"1":0.14,"15":0.06,"5":0.13,"norm":{"1":0.0175,"15":0.0075,"5":0.0163}}}}}}
        2018-12-05T01:26:22.445Z        INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":110,"time":{"ms":13}},"total":{"ticks":260,"time":{"ms":15},"value":260},"user":{"ticks":150,"time":{"ms":2}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":5},"info":{"ephemeral_id":"ca87e6a8-0791-494e-befc-31dd3bef8b45","uptime":{"ms":60220}},"memstats":{"gc_next":4194304,"memory_alloc":3204976,"memory_total":4790960}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":2,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.08,"15":0.06,"5":0.12,"norm":{"1":0.01,"15":0.0075,"5":0.015}}}}}}

steffens · December 5, 2018, 2:08pm

Filebeat is not harvesting any logs: "filebeat":{"harvester":{"open_files":0,"running":0}}

Can you run ls /var/log/nginx/*.log* in your shell?

Rakesh_Thangapandian · December 5, 2018, 3:02pm

</>This was the result of list cmd.

ubuntu@ubuntu-web:~$ ls /var/log/nginx/*.log*
    /var/log/nginx/access.log        /var/log/nginx/access.log.14.gz  /var/log/nginx/access.log.7.gz  /var/log/nginx/error.log.11.gz  /var/log/nginx/error.log.4.gz
    /var/log/nginx/access.log.1      /var/log/nginx/access.log.2.gz   /var/log/nginx/access.log.8.gz  /var/log/nginx/error.log.12.gz  /var/log/nginx/error.log.5.gz
    /var/log/nginx/access.log.10.gz  /var/log/nginx/access.log.3.gz   /var/log/nginx/access.log.9.gz  /var/log/nginx/error.log.13.gz  /var/log/nginx/error.log.6.gz
    /var/log/nginx/access.log.11.gz  /var/log/nginx/access.log.4.gz   /var/log/nginx/error.log        /var/log/nginx/error.log.14.gz  /var/log/nginx/error.log.7.gz
    /var/log/nginx/access.log.12.gz  /var/log/nginx/access.log.5.gz   /var/log/nginx/error.log.1      /var/log/nginx/error.log.2.gz   /var/log/nginx/error.log.8.gz
    /var/log/nginx/access.log.13.gz  /var/log/nginx/access.log.6.gz   /var/log/nginx/error.log.10.gz  /var/log/nginx/error.log.3.gz   /var/log/nginx/error.log.9.gz

I guess mounting nginx log files directory inside filebeat container would solve the issue

steffens · December 5, 2018, 3:47pm

I guess mounting nginx log files directory inside filebeat container would solve the issue

Yep, this would help. Docker containers isolate disks/volumes. Without explicitly mounting the right directories there is nothing for filebeat to read.

Rakesh_Thangapandian · December 6, 2018, 8:16am

Yeah, Issue Resolved...
Thank you So much @steffens

system · January 3, 2019, 8:16am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.