I have Mailcow-dockerize envirment and need to send postfix container logs (Basically maillogs) to external host ELK stack.
ELK stack is working fine.
When I installed filebeat on my mailcow setup and create filebeat yml as below.
#########
# Filebeat configuration file
filebeat.inputs:
- type: container
enabled: true
paths:
- /var/lib/docker/containers/*/*.log
processors:
- add_docker_metadata: ~
# Elasticsearch output
output.elasticsearch:
hosts: ["172.20.111.199:9200"]
# Uncomment and set the following if security is enabled
# username: "elastic"
# password: "<ELK_PASSWORD>"
pipeline: "postfix-pipeline"
index: "mailcow-postfix-%{+yyyy.MM.dd}"
# Logging configuration
logging.level: info
logging.to_files: true
logging.files:
path: /var/log/filebeat
name: filebeat
keepfiles: 7
permissions: 0644
# Required for filebeat modules
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
# Processors
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
# Ingest Node Pipeline
setup.template:
name: "postfix"
pattern: "postfix-*"
overwrite: true`
#########
But I received some error logs in filebeat logs.
As follows,`
#cat /var/log/filebeat/filebeat-20240712-3.ndjson
{"log.level":"info","@timestamp":"2024-07-12T10:41:54.131+0530","log.logger":"monitoring","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logTotals","file.name":"log/log.go","file.line":195},"message":"Total metrics","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpu":{"id":"filebeat.service","stats":{"periods":0,"throttled":{"ns":0,"periods":0}}},"memory":{"id":"filebeat.service","mem":{"usage":{"bytes":39604224}}}},"cpu":{"system":{"ticks":30,"time":{"ms":30}},"total":{"ticks":80,"time":{"ms":80},"value":80},"user":{"ticks":50,"time":{"ms":50}}},"handles":{"limit":{"hard":524288,"soft":524288},"open":17},"info":{"ephemeral_id":"c47d6a44-5d30-4c0f-9aeb-f6aaaed9bd65","name":"filebeat","uptime":{"ms":31},"version":"8.14.3"},"memstats":{"gc_next":32900848,"memory_alloc":19766744,"memory_sys":46241032,"memory_total":39104824,"rss":108920832},"runtime":{"goroutines":33}},"filebeat":{"events":{"active":0,"added":0,"done":0},"harvester":{"closed":0,"open_files":0,"running":0,"skipped":0,"started":0},"input":{"log":{"files":{"renamed":0,"truncated":0}}}},"libbeat":{"config":{"module":{"running":0,"starts":0,"stops":0},"reloads":0,"scans":0},"output":{"batches":{"split":0},"events":{"acked":0,"active":0,"batches":0,"dropped":0,"duplicates":0,"failed":0,"toomany":0,"total":0},"read":{"bytes":0,"errors":0},"type":"elasticsearch","write":{"bytes":0,"errors":0,"latency":{"histogram":{"count":0,"max":0,"mean":0,"median":0,"min":0,"p75":0,"p95":0,"p99":0,"p999":0,"stddev":0}}}},"pipeline":{"clients":0,"events":{"active":0,"dropped":0,"failed":0,"filtered":0,"published":0,"retry":0,"total":0},"queue":{"acked":0,"max_events":3200}}},"processor":{"add_host_metadata":{"fqdn_lookup_failed":0}},"registrar":{"states":{"cleanup":0,"current":0,"update":0},"writes":{"fail":0,"success":0,"total":0}},"system":{"cpu":{"cores":8},"load":{"1":0,"15":0,"5":0,"norm":{"1":0,"15":0,"5":0}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2024-07-12T10:41:54.131+0530","log.logger":"monitoring","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logTotals","file.name":"log/log.go","file.line":196},"message":"Uptime: 28.736987ms","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-07-12T10:41:54.131+0530","log.logger":"monitoring","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).snapshotLoop","file.name":"log/log.go","file.line":163},"message":"Stopping metrics logging.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-07-12T10:41:54.131+0530","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch","file.name":"instance/beat.go","file.line":537},"message":"filebeat stopped.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-07-12T10:41:54.133+0530","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.handleError","file.name":"instance/beat.go","file.line":1345},"message":"Exiting: Failed to start crawler: starting input failed: error while initializing input: Error creating input. No such input type exist: 'docker'","service.name":"filebeat","ecs.version":"1.6.0"}`
What is the issue?
Note: Filebeat start and running.