Hello,
I'm trying to deploy filebeat on Kubernetes that is configured with running CRI-O container runtime environment. Deploying the filebeat as in https://github.com/elastic/beats/blob/master/deploy/kubernetes/filebeat-kubernetes.yaml could not find the container log files.
So for cri-o based environment I have tried modifying the container log path for filebeat as in https://github.com/elastic/beats/issues/8985#issuecomment-437271804
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
namespace: kube-system
labels:
k8s-app: filebeat
data:
filebeat.yml: |-
filebeat.config:
modules:
path: ${path.config}/modules.d/*.yml
# Reload module configs as they change:
reload.enabled: false
filebeat.autodiscover:
providers:
- type: kubernetes
include_pod_uid: true
in_cluster: true
hints.enabled: true
include_annotations: '*'
templates:
- condition.regexp:
kubernetes.container.name: '.+'
config:
input:
type: container
json.keys_under_root: false
json.add_error_key: false
json.ignore_decoding_error: true
combine_partial: true
cri.parse_flags: true
cri.force: true
containers:
path: "/var/log/crio/pods/${data.kubernetes.pod.uid}/"
ids:
- "${data.kubernetes.container.name}"
processors:
- add_cloud_metadata: ~
- add_kubernetes_metadata:
in_cluster: true
output.logstash:
hosts: ['logstash:5044']
FileBeat Logs
2020-02-12T17:12:54.743Z INFO [autodiscover] autodiscover/autodiscover.go:105 Starting autodiscover manager 2020-02-12T17:12:54.845Z INFO log/input.go:152 Configured paths: [/var/lib/docker/containers/2c0aaeb30c01eb4cc7a187e4fa59f8777fce9b9947cad18c688cd5b43dcb655e/*-json.log] 2020-02-12T17:12:54.845Z INFO input/input.go:114 Starting input of type: container; ID: 15337647248318266633 2020-02-12T17:12:54.845Z INFO log/input.go:152 Configured paths: [/var/lib/docker/containers/ee3e3a0f173d0e39b741cc8b94b9f5e6843061418e7845b77981fe0a02bf6739/*-json.log] 2020-02-12T17:12:54.845Z INFO input/input.go:114 Starting input of type: container; ID: 11194825261634314163 2020-02-12T17:12:54.846Z INFO log/input.go:152 Configured paths: [/var/lib/docker/containers/a9cbc5469c305d5bcb2e5f6c1d50271314798075b9490fa1a1951a9d2b2a4ae3/*-json.log] 2020-02-12T17:12:54.846Z INFO input/input.go:114 Starting input of type: container; ID: 7915459815237518958 2020-02-12T17:12:54.846Z INFO log/input.go:152 Configured paths: [/var/lib/docker/containers/dfb185c4e6d39d5dd371f27422f97d118bf9d307e6e69a49c0a3cb3be0ef30b3/*-json.log] 2020-02-12T17:12:54.846Z INFO input/input.go:114 Starting input of type: container; ID: 14976356031607412965 2020-02-12T17:12:54.847Z INFO log/input.go:152 Configured paths: [/var/lib/docker/containers/60f89f881ec85239470d900d7e5e53b3ad4fcf59ec46879e6c9c656a5bf4eedf/*-json.log] 2020-02-12T17:12:54.847Z INFO input/input.go:114 Starting input of type: container; ID: 9390979353581095752 2020-02-12T17:12:54.847Z INFO log/input.go:152 Configured paths: [/var/lib/docker/containers/7ff92e3a3b7d8b7082db39bc68f13b8cf01c3f7ee8aa72eb7fc584bd92b27886/*-json.log] 2020-02-12T17:12:54.847Z INFO input/input.go:114 Starting input of type: container; ID: 10588676206953125327 2020-02-12T17:12:54.848Z INFO log/input.go:152 Configured paths: [/var/lib/docker/containers/2d4eda33c94114f0654989dcd64643dcfc4e9929205de00eebe4a45ba20c06e1/*-json.log] 2020-02-12T17:12:54.848Z INFO input/input.go:114 Starting input of type: container; ID: 12414274566107568665 2020-02-12T17:12:54.848Z INFO log/input.go:152 Configured paths: [/var/lib/docker/containers/19255798918a11a12375f6c5b677fccaf13afdb026846d02c6542dd11aa2a0bb/*-json.log] 2020-02-12T17:12:54.848Z INFO input/input.go:114 Starting input of type: container; ID: 17154145792288559585 2020-02-12T17:12:54.849Z INFO log/input.go:152 Configured paths: [/var/lib/docker/containers/262456cd7d6d9d2e1712090a37817d34f8f6dd9289f675f66f80f875c5689fec/*-json.log] 2020-02-12T17:12:54.849Z INFO input/input.go:114 Starting input of type: container; ID: 11997798302486402131 2020-02-12T17:12:54.849Z INFO log/input.go:152 Configured paths: [/var/lib/docker/containers/fd9c3ef2cb858b07f79d48508bc2899ebcbab1674466f1efbcd9be1385ee252a/*-json.log] 2020-02-12T17:12:54.849Z INFO input/input.go:114 Starting input of type: container; ID: 9689899642930122113 2020-02-12T17:12:54.850Z INFO log/input.go:152 Configured paths: [/var/lib/docker/containers/c23d93f667f2476c10dfd34409e9ffc68eac0de91a64d6fde359384ac3be7b85/*-json.log] 2020-02-12T17:12:54.850Z INFO input/input.go:114 Starting input of type: container; ID: 8237671109160429285 2020-02-12T17:12:54.850Z INFO log/input.go:152 Configured paths: [/var/lib/docker/containers/d507f69c1a147410885f94c7a40297386426448037c50780ddad8d5eff51e8aa/*-json.log] 2020-02-12T17:12:54.850Z INFO input/input.go:114 Starting input of type: container; ID: 5030918935259973164 2020-02-12T17:12:54.851Z INFO log/input.go:152 Configured paths: [/var/lib/docker/containers/72361a57dda867b4819b08394d2d9c3116a64fb2085a691123dc863f84205f74/*-json.log] 2020-02-12T17:12:54.851Z INFO input/input.go:114 Starting input of type: container; ID: 15961752709069961090 2020-02-12T17:12:54.851Z INFO log/input.go:152 Configured paths: [/var/lib/docker/containers/ffa1c77cace5414c84175573acb77e7ee6a1ae2e6d6f78941fd3bf5e4c1229d8/*-json.log] 2020-02-12T17:12:54.851Z INFO input/input.go:114 Starting input of type: container; ID: 3494905796133945119 2020-02-12T17:12:54.852Z INFO log/input.go:152 Configured paths: [/var/lib/docker/containers/79143c8c234b985a1e98c32954ab5c0344779005901ca59f5758a99f55cfae28/*-json.log] 2020-02-12T17:12:54.852Z INFO input/input.go:114 Starting input of type: container; ID: 17402523985284215498
`
but it's still looking at docker path [/var/lib/docker/] and it's neither harvesting the log files nor forwarding them to logstash ?
Appreciate any help
Thanks.