Hey, I am trying to ship logs from kubernates containers.
Any idea how to connect kubernates with filebeat?
How to find the kubernetes container logs?
Where are your log files persisted? In each container, share volume, ...?
Note: I don't know much details about kubernetes ...
Hi nanshan,
The kubernetes containers stdout/stderr log will be exported into /var/log/containers/ in kubernetes node.
So you can use filebeat to monitor container logs under /var/log/containers/:
Here is an example for monitoring kafka container log:
filebeat:
prospectors:
-
paths:
- "/var/log/containers/kafka*.log"
document_type: kube-logs
output:
logstash:
hosts: [${LOGSTASH_HOSTS:'logstash:5043'}]
loadbalance: true
timeout: 15
Hi Lei,
Is it default to export the logs to /var/log/containers?
on the kubernates node,
cd /var/lib/docker/containers/
ls -l
total 60 drwx------ 3 root root 4096 Jul 27 01:52 0e361b43fc13416243b81e192bed7a50090fc0b1cb2c9f642b34f271028a9756 drwx------ 3 root root 4096 Jul 27 01:52 0ff70dd3de991f6de7f0a50b75b49532cfdfb8e78e68eb2c5e1f557e695a570f drwx------ 2 root root 4096 Jul 27 02:05 17470697c6da770ad77d60961c1f8e42ec33950ed291ba68ccbb35ac4cd0067e drwx------ 2 root root 4096 Jul 27 02:04 2f61e9273359b043d4ff6cd02af97f5b3c93b77d4825995d1592134a6ae54b60 drwx------ 3 root root 4096 Jul 27 01:52 30a4f10cab6c782e14cba4c32fad4b6bad6fac4e5fb65559e77d7093d7f92824 drwx------ 2 root root 4096 Jul 27 02:04 500098b4d6efe903471749490227610c52bf7758272277702de0f02f53e66070 drwx------ 2 root root 4096 Jul 27 01:53 54fe4038b1eb2f519c935354c94d3f050ac9f1bef899404ae87f6f9269dc1837 drwx------ 3 root root 4096 Jul 27 01:52 7d1af600555d2c56c1eff8c47a1706432bf0a2040f2efb14dea472f356bbef06 drwx------ 2 root root 4096 Jul 27 01:54 83a859fba0208404109a8117caed9f2f97e1d2b89e0b0759d79a6100da10ee14 drwx------ 2 root root 4096 Jul 27 01:54 a76d4697ed01f972a01b740c315d411bae1dcc0e1fc4996fb462a78c5018dd30 drwx------ 2 root root 4096 Jul 27 02:05 b8e4a04d50c83310cb0cc7dda1637b0d2b235f22d69dc4376702ab35cf4b2e5c drwx------ 3 root root 4096 Jul 27 02:05 b9f03a91a8aebd3147d9573d47c98a5c39137f59b34c6a022122b01d78612e17 drwx------ 3 root root 4096 Jul 27 01:52 beba112c1c385f297ec7754ab975b257c5ab35b90f93e227fc48bfb20b9a1722 drwx------ 2 root root 4096 Jul 27 02:05 ea84469666da4a588c31d4eabfb33e5c7432189eb61a6a72eed73bfd7071003e drwx------ 3 root root 4096 Jul 27 02:04 f0a8ea4c77176e510cd66c14923a9f08bba2cf4fca24ce3702f7cfd7f9e4567f
all of these long name are container id
How to set up like you mentioned 'kafka*.log" ?
if you look in /var/log/containers the files are symbolic links (by pod name) to the container logs in /var/lib/docker/containers
Yes, matt is right. You can find all the logs under /var/log/containers.
paths:
- "/var/log/containers/kafka*.log"
means that filebeat will monitor all the kafka*.log in /var/log/containers
lrwxrwxrwx 1 root root 165 Jul 27 03:10 dcdr-z8aem_default_POD-c13f6276a7a8737536a8b76f30fa0e4da3b7b7051610a5525f4606e48696340c.log -> /var/lib/docker/containers/c13f6276a7a8737536a8b76f30fa0e4da3b7b7051610a5525f4606e48696340c/c13f6276a7a8737536a8b76f30fa0e4da3b7b7051610a5525f4606e48696340c-json.log lrwxrwxrwx 1 root root 165 Jul 27 03:10 dcdr-z8aem_default_dcdr-5e5cc39669f075425e8ee87b201cc84e0d217785c9c5616b8c65316095d35833.log -> /var/lib/docker/containers/5e5cc39669f075425e8ee87b201cc84e0d217785c9c5616b8c65316095d35833/5e5cc39669f075425e8ee87b201cc84e0d217785c9c5616b8c65316095d35833-json.log lrwxrwxrwx 1 root root 165 Jul 28 02:12 heapster-v1.1.0-1394722715-mupj0_kube-system_POD-ad63dc26e982e73a9f9429d9ac7678115209511369595921eb9b21ab457aa279.log -> /var/lib/docker/containers/ad63dc26e982e73a9f9429d9ac7678115209511369595921eb9b21ab457aa279/ad63dc26e982e73a9f9429d9ac7678115209511369595921eb9b21ab457aa279-json.log lrwxrwxrwx 1 root root 165 Jul 28 02:12 heapster-v1.1.0-1394722715-mupj0_kube-system_heapster-01dcf8872867467afe17c28e4c7c5e1cb767e16f5b2eda1ed149d994039ccf17.log -> /var/lib/docker/containers/01dcf8872867467afe17c28e4c7c5e1cb767e16f5b2eda1ed149d994039ccf17/01dcf8872867467afe17c28e4c7c5e1cb767e16f5b2eda1ed149d994039ccf17-json.log lrwxrwxrwx 1 root root 165 Jul 28 02:13 heapster-v1.1.0-1394722715-mupj0_kube-system_heapster-nanny-3aa7263583e90848c91c774dac90ad609f8cd8aff526fb353b2ac47cfc8cdf61.log -> /var/lib/docker/containers/3aa7263583e90848c91c774dac90ad609f8cd8aff526fb353b2ac47cfc8cdf61/3aa7263583e90848c91c774dac90ad609f8cd8aff526fb353b2ac47cfc8cdf61-json.log lrwxrwxrwx 1 root root 165 Aug 1 17:26 sync-api-2522378034-359cv_default_POD-0a0eb6b35c3210390468dceb7dde674eb986349d1368851b5c08f36819e2ac3a.log -> /var/lib/docker/containers/0a0eb6b35c3210390468dceb7dde674eb986349d1368851b5c08f36819e2ac3a/0a0eb6b35c3210390468dceb7dde674eb986349d1368851b5c08f36819e2ac3a-json.log lrwxrwxrwx 1 root root 165 Aug 1 17:26 sync-api-2522378034-359cv_default_sync-api-09b1b883f4f680643c98a2289d2874208e052ea4740ed82a6d38098655ab2c41.log -> /var/lib/docker/containers/09b1b883f4f680643c98a2289d2874208e052ea4740ed82a6d38098655ab2c41/09b1b883f4f680643c98a2289d2874208e052ea4740ed82a6d38098655ab2c41-json.log
I can see the softlink but the link names ares random id instead of actually logs names like sync-api* like kafka*.log
How to change the name?
Just saw that some of the above are symlinks. Be aware that in 5.0 by default symlinks are not followed anymore to prevent potential duplicate data. We are thinking of introducing a config option to enable it. See https://github.com/elastic/beats/issues/1686 If you need symlinks it is best to open a feature request in the beats repo for it.
You don't need to change the name, for example, below setting will allow filebeat(version 1.2.3) to fetch the stdout logs of container sync-api.
paths:
- "/var/log/containers/sync-api*.log"
@ruflin thx a lot for mentioning that
This topic was automatically closed after 21 days. New replies are no longer allowed.