Syslog harvester does not start. Running Filebeat 7.3.1 on Docker

glauber.ferreira · September 27, 2019, 12:57pm

I am running Elastic Stack 7.3.1 on Docker (Elastic, Kibana, Metricbeat, Filebeat). Filebeat harvesting are being started successfully for Docker container files. But, harvesting for Syslog module are not being started.

This is my filebeat log file. There are four lines starting with Harvester started for file; all of them related to docker containers. But there is not a line indicating the start of harvester for syslog files.

2019-09-27T12:00:23.788Z        INFO    instance/beat.go:606    Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2019-09-27T12:00:23.789Z        INFO    instance/beat.go:614    Beat ID: ef9dd9a2-06ad-4358-88e8-cd91c34a9a08
2019-09-27T12:00:23.799Z        INFO    [seccomp]       seccomp/seccomp.go:124  Syscall filter successfully installed
2019-09-27T12:00:23.835Z        INFO    instance/beat.go:292    Setup Beat: filebeat; Version: 7.3.1
2019-09-27T12:00:23.835Z        INFO    [index-management]      idxmgmt/std.go:178      Set output.elasticsearch.index to 'filebeat-7.3.1' as ILM is enabled.
2019-09-27T12:00:23.836Z        INFO    elasticsearch/client.go:170     Elasticsearch url: https://192.168.1.6:9200
2019-09-27T12:00:23.836Z        INFO    [publisher]     pipeline/module.go:97   Beat name: vm-07
2019-09-27T12:00:23.902Z        INFO    beater/filebeat.go:92   Enabled modules/filesets: system (syslog),  ()
2019-09-27T12:00:23.902Z        INFO    instance/beat.go:421    filebeat start running.
2019-09-27T12:00:23.902Z        INFO    [monitoring]    log/log.go:118  Starting metrics logging every 30s
2019-09-27T12:00:23.903Z        INFO    registrar/migrate.go:104        No registry home found. Create: /usr/share/filebeat/data/registry/filebeat
2019-09-27T12:00:23.908Z        INFO    registrar/migrate.go:112        Initialize registry meta file
2019-09-27T12:00:23.910Z        INFO    registrar/registrar.go:108      No registry file found under: /usr/share/filebeat/data/registry/filebeat/data.json. Creating a new registry file.
2019-09-27T12:00:23.911Z        INFO    registrar/registrar.go:145      Loading registrar data from /usr/share/filebeat/data/registry/filebeat/data.json
2019-09-27T12:00:23.911Z        INFO    registrar/registrar.go:152      States Loaded from registrar: 0
2019-09-27T12:00:23.911Z        INFO    crawler/crawler.go:72   Loading Inputs: 1
2019-09-27T12:00:23.918Z        INFO    log/input.go:148        Configured paths: [/var/log/messages* /var/log/syslog*]
2019-09-27T12:00:23.918Z        INFO    input/input.go:114      Starting input of type: log; ID: 6766254637019430523
2019-09-27T12:00:23.918Z        INFO    crawler/crawler.go:106  Loading and starting Inputs completed. Enabled inputs: 1
2019-09-27T12:00:23.918Z        WARN    [cfgwarn]       docker/docker.go:57     BETA: The docker autodiscover is beta
2019-09-27T12:00:23.918Z        INFO    cfgfile/reload.go:171   Config reloader started
2019-09-27T12:00:23.919Z        INFO    cfgfile/reload.go:226   Loading of config files completed.
2019-09-27T12:00:23.938Z        INFO    [autodiscover]  autodiscover/autodiscover.go:105        Starting autodiscover manager
2019-09-27T12:00:23.941Z        INFO    log/input.go:148        Configured paths: [/var/lib/docker/containers/c3bfe253216cdda7bd2fb3712be1b3cd2baac4918fc41d5c0b71f4975fcb0e2d/*-json.log]
2019-09-27T12:00:23.941Z        INFO    input/input.go:114      Starting input of type: container; ID: 9980403068538753444
2019-09-27T12:00:23.942Z        INFO    log/harvester.go:253    Harvester started for file: /var/lib/docker/containers/c3bfe253216cdda7bd2fb3712be1b3cd2baac4918fc41d5c0b71f4975fcb0e2d/c3bfe253216cdda7bd2fb3712be1b3cd2baac4918fc41d5c0b71f4975fcb0e2d-json.log
2019-09-27T12:00:23.963Z        INFO    log/input.go:148        Configured paths: [/var/lib/docker/containers/19b2a9c435da1aacadf891900fd2848d103f89d694324c439ab2f145d4c6cc45/*-json.log]
2019-09-27T12:00:23.963Z        INFO    log/input.go:148        Configured paths: [/var/lib/docker/containers/19b2a9c435da1aacadf891900fd2848d103f89d694324c439ab2f145d4c6cc45/*-json.log]
2019-09-27T12:00:23.964Z        INFO    elasticsearch/client.go:170     Elasticsearch url: https://192.168.1.6:9200
2019-09-27T12:00:23.982Z        INFO    elasticsearch/client.go:743     Attempting to connect to Elasticsearch version 7.3.1
2019-09-27T12:00:24.006Z        INFO    input/input.go:114      Starting input of type: container; ID: 16974832933917144666
2019-09-27T12:00:24.006Z        INFO    input/input.go:114      Starting input of type: container; ID: 908475651821469795
2019-09-27T12:00:24.006Z        INFO    log/input.go:148        Configured paths: [/var/lib/docker/containers/d0c0c3a9b3066eb8393a110ca8019c9b0760eb2e076744662065ba295c4b0913/*-json.log]
2019-09-27T12:00:24.006Z        INFO    input/input.go:114      Starting input of type: container; ID: 15115359509441093605
2019-09-27T12:00:24.007Z        INFO    log/harvester.go:253    Harvester started for file: /var/lib/docker/containers/19b2a9c435da1aacadf891900fd2848d103f89d694324c439ab2f145d4c6cc45/19b2a9c435da1aacadf891900fd2848d103f89d694324c439ab2f145d4c6cc45-json.log
2019-09-27T12:00:24.007Z        INFO    log/harvester.go:253    Harvester started for file: /var/lib/docker/containers/19b2a9c435da1aacadf891900fd2848d103f89d694324c439ab2f145d4c6cc45/19b2a9c435da1aacadf891900fd2848d103f89d694324c439ab2f145d4c6cc45-json.log
2019-09-27T12:00:24.007Z        INFO    log/harvester.go:253    Harvester started for file: /var/lib/docker/containers/d0c0c3a9b3066eb8393a110ca8019c9b0760eb2e076744662065ba295c4b0913/d0c0c3a9b3066eb8393a110ca8019c9b0760eb2e076744662065ba295c4b0913-json.log
2019-09-27T12:00:24.147Z        INFO    pipeline/output.go:95   Connecting to backoff(elasticsearch(https://192.168.1.6:9200))
2019-09-27T12:00:24.161Z        INFO    elasticsearch/client.go:743     Attempting to connect to Elasticsearch version 7.3.1
2019-09-27T12:00:24.206Z        INFO    [index-management]      idxmgmt/std.go:252      Auto ILM enable success.
2019-09-27T12:00:24.206Z        INFO    [index-management.ilm]  ilm/std.go:134  do not generate ilm policy: exists=true, overwrite=false
2019-09-27T12:00:24.206Z        INFO    [index-management]      idxmgmt/std.go:265      ILM policy successfully loaded.
2019-09-27T12:00:24.208Z        INFO    [index-management]      idxmgmt/std.go:300      Write alias successfully generated.
2019-09-27T12:00:24.225Z        INFO    pipeline/output.go:105  Connection to backoff(elasticsearch(https://192.168.1.6:9200)) established

Anyone can help me? I don't know what I am missing.

...

glauber.ferreira · September 27, 2019, 12:59pm

Last line of log...

2019-09-27T12:00:53.911Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":950,"time":{"ms":958}},"total":{"ticks":6360,"time":{"ms":6375},"value":6360},"user":{"ticks":5410,"time":{"ms":5417}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":13},"info":{"ephemeral_id":"c05f9d31-a89b-4f0c-9f80-72dd28101ae6","uptime":{"ms":30137}},"memstats":{"gc_next":58904112,"memory_alloc":39445224,"memory_total":826252240,"rss":104927232},"runtime":{"goroutines":74}},"filebeat":{"events":{"active":4119,"added":46483,"done":42364},"harvester":{"open_files":4,"running":4,"started":4}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"events":{"acked":42360,"active":50,"batches":849,"total":42410},"read":{"bytes":9735265},"type":"elasticsearch","write":{"bytes":80293951}},"pipeline":{"clients":5,"events":{"active":4119,"filtered":4,"published":46476,"retry":50,"total":46483},"queue":{"acked":42360}}},"registrar":{"states":{"current":3,"update":42364},"writes":{"success":852,"total":852}},"system":{"cpu":{"cores":2},"load":{"1":0.35,"15":0.18,"5":0.17,"norm":{"1":0.175,"15":0.09,"5":0.085}}}}}}

The configured paths for Syslog module are correct. The files exist.

ls -l /var/log/syslog*
-rw-r----- 1 syslog adm  2870704 set 27 09:34 /var/log/syslog
-rw-r----- 1 syslog adm 21769097 set 27 06:25 /var/log/syslog.1
-rw-r----- 1 syslog adm   977882 set 26 06:25 /var/log/syslog.2.gz
-rw-r----- 1 syslog adm   977450 set 25 06:25 /var/log/syslog.3.gz

This is my filebeat configuration file:

filebeat.modules:
- module: system
# Syslog
  syslog:
    enabled: true
# Authorization logs
  auth:
    enabled: false
filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true
filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false
output.elasticsearch:
  hosts: '${ELASTICSEARCH_HOSTS:}'
  username: '${ELASTICSEARCH_USERNAME:}'
  password: '${ELASTICSEARCH_PASSWORD:}'
  ssl.certificate_authorities: '${ELASTICSEARCH_SSL_CERTIFICATE_AUTHORITIES:}'
setup.template.enabled: false
setup.ilm.check_exists: false
setup.ilm.overwrite: false

glauber.ferreira · September 30, 2019, 7:58pm

Solved. I need to mount syslog host files in order to docker container access them. See https://discuss.elastic.co/t/file-beats-in-docker-how-to-access-host-logs/127313/2

system · October 28, 2019, 7:58pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.