Filebeat docker on mac connecting to logstash deployed in Openshift

Hi all,

I am newbie to elk and filebeat. I have deployed EK(version 7.6.1) operator and logstash(version 7.6.1) in openshift. This scenaio works fine when I upload logs file to logstash. Now I am want to use filebeat to send data to logstash. I have Filebeat running as a docker container on Mac. On startup of filebeat docker I was expecting to see some logging with regards to connection success/failure to logstash. Also on startup I am mounting local log files to /usr/share/filebeaat/data folder but I don't see any processing logs. Any advice, much appreciated.

Thanks
Raj

Filebeat.yml

ilebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

output.logstash:
  hosts: '${LOGSTASH_HOSTS}'
  timeout: 15
  ssl:
    certificate_authorities:
      - /etc/pki/tls/certs/logstash-beats.crt  

inputs:
- type: log
  paths:
    - /usr/share/filebeat/data/*.xml

filebeat.modules:
- module: logstash

logging.level: DEBUG

Start filebeat

$ docker run -d --rm --name=filebeat --user=root --volume="$(pwd)/development/filebeat.docker.yml:/usr/share/filebeat/filebeat.yml:ro" --volume="$(pwd)/development/tls.crt:/etc/pki/tls/certs/logstash-beats.crt:ro" --volume="$(pwd)/development/facetlogs:/usr/share/filebeat/data" docker.elastic.co/beats/filebeat:7.6.1 filebeat -e -strict.perms=false -E LOGSTASH_HOSTS=["logstash-logging.ip.dev.com"]

Filebeat startup logs

2020-03-20T14:45:00.653Z    INFO    instance/beat.go:622    Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2020-03-20T14:45:00.653Z    DEBUG   [beat]  instance/beat.go:674    Beat metadata path: /usr/share/filebeat/data/meta.json
2020-03-20T14:45:00.656Z    INFO    instance/beat.go:630    Beat ID: 055a89a5-9bb7-41fe-bb3f-d6869d2d9be0
2020-03-20T14:45:00.658Z    INFO    [beat]  instance/beat.go:958    Beat info   {"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "055a89a5-9bb7-41fe-bb3f-d6869d2d9be0"}}}
2020-03-20T14:45:00.658Z    INFO    [beat]  instance/beat.go:967    Build info  {"system_info": {"build": {"commit": "c1c49432bdc53563e63e9d684ca3e9843626e448", "libbeat": "7.6.1", "time": "2020-02-28T23:12:26.000Z", "version": "7.6.1"}}}
2020-03-20T14:45:00.659Z    INFO    instance/beat.go:298    Setup Beat: filebeat; Version: 7.6.1
2020-03-20T14:45:00.659Z    DEBUG   [beat]  instance/beat.go:324    Initializing output plugins
2020-03-20T14:45:00.662Z    DEBUG   [tls]   tlscommon/tls.go:155    successfully loaded CA certificate: /etc/pki/tls/certs/logstash-beats.crt
2020-03-20T14:45:00.662Z    DEBUG   [publisher] pipeline/consumer.go:137    start pipeline event consumer
2020-03-20T14:45:00.662Z    INFO    [publisher] pipeline/module.go:110  Beat name: 2d5e05bd6726
2020-03-20T14:45:00.664Z    INFO    beater/filebeat.go:92   Enabled modules/filesets: logstash (log, slowlog),  ()
2020-03-20T14:45:00.664Z    WARN    beater/filebeat.go:152  Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-03-20T14:45:00.665Z    INFO    [monitoring]    log/log.go:118  Starting metrics logging every 30s
2020-03-20T14:45:00.665Z    INFO    instance/beat.go:439    filebeat start running.
2020-03-20T14:45:00.665Z    WARN    beater/filebeat.go:181  Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-03-20T14:45:00.666Z    DEBUG   [test]  registrar/migrate.go:159    isFile(/usr/share/filebeat/data/registry) -> false
2020-03-20T14:45:00.666Z    DEBUG   [test]  registrar/migrate.go:159    isFile() -> false
2020-03-20T14:45:00.667Z    DEBUG   [test]  registrar/migrate.go:152    isDir(/usr/share/filebeat/data/registry/filebeat) -> true
2020-03-20T14:45:00.668Z    DEBUG   [test]  registrar/migrate.go:159    isFile(/usr/share/filebeat/data/registry/filebeat/meta.json) -> true
2020-03-20T14:45:00.670Z    DEBUG   [registrar] registrar/migrate.go:51 Registry type '0' found
2020-03-20T14:45:00.671Z    DEBUG   [registrar] registrar/registrar.go:125  Registry file set to: /usr/share/filebeat/data/registry/filebeat/data.json
2020-03-20T14:45:00.672Z    INFO    registrar/registrar.go:145  Loading registrar data from /usr/share/filebeat/data/registry/filebeat/data.json
2020-03-20T14:45:00.673Z    INFO    registrar/registrar.go:152  States Loaded from registrar: 0
2020-03-20T14:45:00.673Z    WARN    beater/filebeat.go:368  Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-03-20T14:45:00.673Z    INFO    crawler/crawler.go:72   Loading Inputs: 2
2020-03-20T14:45:00.674Z    DEBUG   [registrar] registrar/registrar.go:278  Starting Registrar
2020-03-20T14:45:00.674Z    DEBUG   [conditions]    conditions/conditions.go:98 New condition regexp: map[]
2020-03-20T14:45:00.674Z    DEBUG   [conditions]    conditions/conditions.go:98 New condition !regexp: map[]
2020-03-20T14:45:00.674Z    DEBUG   [processors]    processors/processor.go:101 Generated new processors: add_locale=[format=offset], condition=!regexp: map[]
2020-03-20T14:45:00.674Z    DEBUG   [input] log/config.go:204   recursive glob enabled
2020-03-20T14:45:00.674Z    DEBUG   [input] log/input.go:164    exclude_files: [(?-s:.)gz(?-m:$)]. Number of stats: 0
2020-03-20T14:45:00.675Z    DEBUG   [input] log/input.go:185    input with previous states loaded: 0
2020-03-20T14:45:00.675Z    INFO    log/input.go:152    Configured paths: [/var/log/logstash/logstash-plain*.log]
2020-03-20T14:45:00.675Z    INFO    input/input.go:114  Starting input of type: log; ID: 8601861250991668688
2020-03-20T14:45:00.675Z    DEBUG   [input] log/input.go:191    Start next scan
2020-03-20T14:45:00.675Z    DEBUG   [input] log/input.go:212    input states cleaned up. Before: 0, After: 0, Pending: 0
2020-03-20T14:45:00.675Z    DEBUG   [conditions]    conditions/conditions.go:98 New condition regexp: map[]
2020-03-20T14:45:00.674Z    DEBUG   [conditions]    conditions/conditions.go:98 New condition !regexp: map[]
2020-03-20T14:45:00.674Z    DEBUG   [processors]    processors/processor.go:101 Generated new processors: add_locale=[format=offset], condition=!regexp: map[]
2020-03-20T14:45:00.674Z    DEBUG   [input] log/config.go:204   recursive glob enabled
2020-03-20T14:45:00.674Z    DEBUG   [input] log/input.go:164    exclude_files: [(?-s:.)gz(?-m:$)]. Number of stats: 0
2020-03-20T14:45:00.675Z    DEBUG   [input] log/input.go:185    input with previous states loaded: 0
2020-03-20T14:45:00.675Z    INFO    log/input.go:152    Configured paths: [/var/log/logstash/logstash-plain*.log]
2020-03-20T14:45:00.675Z    INFO    input/input.go:114  Starting input of type: log; ID: 8601861250991668688
2020-03-20T14:45:00.675Z    DEBUG   [input] log/input.go:191    Start next scan
2020-03-20T14:45:00.675Z    DEBUG   [input] log/input.go:212    input states cleaned up. Before: 0, After: 0, Pending: 0
2020-03-20T14:45:00.675Z    DEBUG   [conditions]    conditions/conditions.go:98 New condition regexp: map[]
2020-03-20T14:45:00.675Z    DEBUG   [conditions]    conditions/conditions.go:98 New condition !regexp: map[]
2020-03-20T14:45:00.675Z    DEBUG   [processors]    processors/processor.go:101 Generated new processors: add_locale=[format=offset], condition=!regexp: map[]
2020-03-20T14:45:00.675Z    DEBUG   [input] log/config.go:204   recursive glob enabled
2020-03-20T14:45:00.676Z    DEBUG   [input] log/input.go:164    exclude_files: [(?-s:.)gz(?-m:$)]. Number of stats: 0
2020-03-20T14:45:00.676Z    DEBUG   [input] log/input.go:185    input with previous states loaded: 0
2020-03-20T14:45:00.676Z    INFO    log/input.go:152    Configured paths: [/var/log/logstash/logstash-slowlog-plain*.log]
2020-03-20T14:45:00.676Z    INFO    input/input.go:114  Starting input of type: log; ID: 16452081253528920147
2020-03-20T14:45:00.676Z    DEBUG   [cfgfile]   cfgfile/reload.go:133   Checking module configs from: /usr/share/filebeat/modules.d/*.yml
2020-03-20T14:45:00.676Z    DEBUG   [input] log/input.go:191    Start next scan
2020-03-20T14:45:00.676Z    DEBUG   [input] log/input.go:212    input states cleaned up. Before: 0, After: 0, Pending: 0
2020-03-20T14:45:00.676Z    DEBUG   [cfgfile]   cfgfile/reload.go:147   Number of module configs found: 0
2020-03-20T14:45:00.676Z    INFO    crawler/crawler.go:106  Loading and starting Inputs completed. Enabled inputs: 2
2020-03-20T14:45:00.676Z    INFO    cfgfile/reload.go:171   Config reloader started
2020-03-20T14:45:00.677Z    DEBUG   [cfgfile]   cfgfile/reload.go:197   Scan for new config files
2020-03-20T14:45:00.677Z    DEBUG   [cfgfile]   cfgfile/reload.go:216   Number of module configs found: 0
2020-03-20T14:45:00.677Z    DEBUG   [reload]    cfgfile/list.go:62  Starting reload procedure, current runners: 0
2020-03-20T14:45:00.677Z    DEBUG   [reload]    cfgfile/list.go:80  Start list: 0, Stop list: 0
2020-03-20T14:45:00.677Z    INFO    cfgfile/reload.go:226   Loading of config files completed.
2020-03-20T14:45:10.679Z    DEBUG   [input] input/input.go:152  Run input
2020-03-20T14:45:10.679Z    DEBUG   [input] log/input.go:191    Start next scan
2020-03-20T14:45:10.679Z    DEBUG   [input] log/input.go:212    input states cleaned up. Before: 0, After: 0, Pending: 0
2020-03-20T14:45:10.679Z    DEBUG   [input] input/input.go:152  Run input
2020-03-20T14:45:10.679Z    DEBUG   [input] log/input.go:191    Start next scan
2020-03-20T14:45:10.679Z    DEBUG   [input] log/input.go:212    input states cleaned up. Before: 0, After: 0, Pending: 0
2020-03-20T14:45:20.645Z    DEBUG   [input] input/input.go:152  Run input
2020-03-20T14:45:20.645Z    DEBUG   [input] log/input.go:191    Start next scan
2020-03-20T14:45:20.645Z    DEBUG   [input] log/input.go:212    input states cleaned up. Before: 0, After: 0, Pending: 0
2020-03-20T14:45:20.645Z    DEBUG   [input] input/input.go:152  Run input
2020-03-20T14:45:20.646Z    DEBUG   [input] log/input.go:191    Start next scan
2020-03-20T14:45:20.646Z    DEBUG   [input] log/input.go:212    input states cleaned up. Before: 0, After: 0, Pending: 0
2020-03-20T14:45:30.639Z    INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":40,"time":{"ms":48}},"total":{"ticks":100,"time":{"ms":108},"value":100},"user":{"ticks":60,"time":{"ms":60}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":6},"info":{"ephemeral_id":"1c95e057-60a4-4d75-b15e-4d4b184c9b54","uptime":{"ms":30046}},"memstats":{"gc_next":8162320,"memory_alloc":5191688,"memory_total":13367280,"rss":48828416},"runtime":{"goroutines":28}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"type":"logstash"},"pipeline":{"clients":2,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":6},"load":{"1":0.04,"15":0,"5":0.04,"norm":{"1":0.0067,"15":0,"5":0.0067}}}}}}

Below command shows "filebeat stopped." I have only one docker instance running.

$ docker exec filebeat ./filebeat -c filebeat.yml -e -d "*"
2020-03-23T13:42:19.810Z	INFO	instance/beat.go:622	Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2020-03-23T13:42:19.810Z	DEBUG	[beat]	instance/beat.go:674	Beat metadata path: /usr/share/filebeat/data/meta.json
2020-03-23T13:42:19.813Z	INFO	instance/beat.go:630	Beat ID: 055a89a5-9bb7-41fe-bb3f-d6869d2d9be0
2020-03-23T13:42:19.814Z	INFO	instance/beat.go:380	filebeat stopped.
2020-03-23T13:42:19.814Z	ERROR	instance/beat.go:933	Exiting: data path already locked by another beat
Exiting: data path already locked by another beat