Hi,
I have set up log collection from docker containers running on EC2 machines to elastic cloud using file beat. But I don't find the logs on kibana.
Note: I have setup my own indexes and ILM and templates. I also find the logs of filebeat don't specify any issue as such. Pasting the configuration of filebeat and the indexes related snippets.
filebeat.inputs:
#------------------------------ Container input --------------------------------
- type: container
enabled: true
# Paths for container logs that should be crawled and fetched.
paths:
-/var/lib/docker/containers/*/*.log
# Configure stream to filter to a specific stream: stdout, stderr or all (default)
stream: all
# ================================= Processors =================================
processors:
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_host_metadata: ~
#
# =============================== Elastic Cloud ================================
cloud.id: "tazapay-dev:xxx"
cloud.auth: "elastic:xxx"
# ====================== Index Lifecycle Management (ILM) ======================
setup.ilm.enabled: false
setup.template.enabled: false
setup.template.name: "core-template"
setup.template.pattern: "core-*"
output.elasticsearch:
indices:
- index: "core"
TEMPLATE :
PUT _template/core-template
{
"index_patterns": ["core-*"],
"settings": {
"number_of_shards": 1,
"index.lifecycle.name": "core",
"index.lifecycle.rollover_alias": "core"
}
}
INDEX :
PUT core-000001
{
"aliases": {
"core":{
"is_write_index": true
}
}
}
FILEBEAT LOGS:
pipeline/module.go:113 Beat name: ip-20-10-1-15
[monitoring] log/log.go:142 Starting metrics logging every 30s
instance/beat.go:473 filebeat start running.
memlog/store.go:119 Loading data file of '/var/lib/filebeat/registry/filebeat' succeeded. Active transaction id=0
memlog/store.go:124 Finished loading transaction log file for '/var/lib/filebeat/registry/filebeat'. Active transaction id=0
[registrar] registrar/registrar.go:109 States Loaded from registrar: 0
[crawler] beater/crawler.go:71 Loading Inputs: 1
[input] log/input.go:164 Configured paths: [/-/var/lib/docker/containers//.log] {"input_id": "08243e87-463a-4814-bed0-823e8438cfe6"}
[crawler] beater/crawler.go:141 Starting input (ID: 3876510880850742928)
[crawler] beater/crawler.go:108 Loading and starting Inputs completed. Enabled inputs: 1