Filebeat:7.10.0 not publishing/pushing logs to single node elasticsearch cluster deployed in kubernetes cluster

nirmal_elastic · February 3, 2021, 1:04pm

I have single node elasticsearch deployed on master node of k8s cluster and it's accessible using nodePort:30339

curl http://xx.xx.xx.xx:30339
{
"name" : "elasticsearch-0",
"cluster_name" : "docker-cluster",
"cluster_uuid" : "z06Zn4p-T3CzUk6hoSiDLg",
"version" : {
"number" : "7.10.0",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "51e9d6f22758d0374a0f3f5c6e8f3a7997850f96",
"build_date" : "2020-11-09T21:30:33.964949Z",
"build_snapshot" : false,
"lucene_version" : "8.7.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}

I have 3 node k8s cluster and deployed filebeat:7.10.0 as daemonset on all nodes including master. It's deployed in discoverable mode and working properly and picked the available container logs. But the logs available in filebeat is not available in elasticsearch and no indexes showing in Kibana.

kubectl -n elasticsearch logs filebeat-lb6wz
2021-02-03T12:24:24.896Z INFO instance/beat.go:645 Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2021-02-03T12:24:24.897Z INFO instance/beat.go:653 Beat ID: 14811f9a-2e7b-4fdf-a007-a873a5fa56c8
2021-02-03T12:24:24.901Z INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed
2021-02-03T12:24:24.901Z INFO [beat] instance/beat.go:981 Beat info {"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "14811f9a-2e7b-4fdf-a007-a873a5fa56c8"}}}
2021-02-03T12:24:24.901Z INFO [beat] instance/beat.go:990 Build info {"system_info": {"build": {"commit": "1428d58cf2ed945441fb2ed03961cafa9e4ad3eb", "libbeat": "7.10.0", "time": "2020-11-09T19:57:04.000Z", "version": "7.10.0"}}}
2021-02-03T12:24:24.901Z INFO [beat] instance/beat.go:993 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.14.7"}}}
2021-02-03T12:24:24.904Z INFO [beat] instance/beat.go:997 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-07-16T11:30:33Z","containerized":true,"name":"dcpprsapp013","ip":["127.0.0.1/8","::1/128","10.77.xx.xx/24","fe80::250:56ff:fe8e:703d/64","172.17.0.1/16","fe80::42:eaff:fef2:af94/64","192.168.115.64/32","fe80::cc00:6bff:febf:4280/64","fe80::18d2:66ff:fedd:d8ae/64","fe80::44ee:bdff:fea9:f926/64","fe80::204e:95ff:fe31:bc08/64","fe80::e8a8:ebff:fe88:bc97/64"],"kernel_version":"3.10.0-1062.1.2.el7.x86_64","mac":["00:50:56:8e:70:3d","02:42:ea:f2:af:94","ce:00:6b:bf:42:80","1a:d2:66:dd:d8:ae","46:ee:bd:a9:f9:26","22:4e:95:31:bc:08","ea:a8:eb:88:bc:97"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":8,"patch":2003,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
2021-02-03T12:24:24.904Z INFO [beat] instance/beat.go:1026 Process info {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2021-02-03T12:24:24.350Z"}}}
2021-02-03T12:24:24.904Z INFO instance/beat.go:299 Setup Beat: filebeat; Version: 7.10.0
2021-02-03T12:24:24.904Z INFO [index-management] idxmgmt/std.go:184 Set output.elasticsearch.index to 'filebeat-7.10.0' as ILM is enabled.
2021-02-03T12:24:24.905Z INFO eslegclient/connection.go:99 elasticsearch url: http://10.77.xx.xx:30339
2021-02-03T12:24:24.905Z INFO [publisher] pipeline/module.go:113 Beat name: dcpprsapp013
2021-02-03T12:24:24.907Z INFO instance/beat.go:455 filebeat start running.
2021-02-03T12:24:24.909Z INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2021-02-03T12:24:24.909Z INFO memlog/store.go:119 Loading data file of '/usr/share/filebeat/data/registry/filebeat' succeeded. Active transaction id=0
2021-02-03T12:24:24.909Z INFO memlog/store.go:124 Finished loading transaction log file for '/usr/share/filebeat/data/registry/filebeat'. Active transaction id=0
2021-02-03T12:24:24.909Z INFO [registrar] registrar/registrar.go:109 States Loaded from registrar: 0
2021-02-03T12:24:24.909Z INFO [crawler] beater/crawler.go:71 Loading Inputs: 0
2021-02-03T12:24:24.909Z INFO [crawler] beater/crawler.go:108 Loading and starting Inputs completed. Enabled inputs: 0
2021-02-03T12:24:24.911Z INFO [autodiscover.pod] kubernetes/util.go:99 kubernetes: Using node dcpprsapp013 provided in the config
2021-02-03T12:24:24.911Z INFO [autodiscover] autodiscover/autodiscover.go:113 Starting autodiscover manager
2021-02-03T12:24:25.012Z INFO log/input.go:157 Configured paths: [/var/log/containers/*9fb4dda333de69326037dc367b11232adec85938501c4b39a168f7677443ff40.log]
2021-02-03T12:24:25.012Z INFO log/input.go:157 Configured paths: [/var/log/containers/*9fb4dda333de69326037dc367b11232adec85938501c4b39a168f7677443ff40.log]
2021-02-03T12:24:25.012Z INFO log/input.go:157 Configured paths: [/var/log/containers/*a91e8afe3ca4085e62bd0b432a3ac2039b6ecc987d086371a5db956396226f51.log]
2021-02-03T12:24:25.013Z INFO log/input.go:157 Configured paths: [/var/log/containers/*a91e8afe3ca4085e62bd0b432a3ac2039b6ecc987d086371a5db956396226f51.log]
2021-02-03T12:24:25.013Z INFO log/input.go:157 Configured paths: [/var/log/containers/*9cdac53e2dcd99614b2c1b140f001d52a1d8fbe091ee4e76913b53fe5a746b75.log]
2021-02-03T12:24:25.013Z INFO log/input.go:157 Configured paths: [/var/log/containers/*9cdac53e2dcd99614b2c1b140f001d52a1d8fbe091ee4e76913b53fe5a746b75.log]
2021-02-03T12:24:25.013Z INFO log/input.go:157 Configured paths: [/var/log/containers/*96d7da17978f85a774bbbf1e20d360b105c287707d5e05a159bb3e95999d324a.log]
2021-02-03T12:24:25.014Z INFO log/input.go:157 Configured paths: [/var/log/containers/*96d7da17978f85a774bbbf1e20d360b105c287707d5e05a159bb3e95999d324a.log]
2021-02-03T12:24:25.014Z INFO log/input.go:157 Configured paths: [/var/log/containers/*2b51e8f4cd9d963ef9d8761275dc2d656b3fd730b0075148ba06fb6c79ed913a.log]
2021-02-03T12:24:25.014Z INFO log/input.go:157 Configured paths: [/var/log/containers/*2b51e8f4cd9d963ef9d8761275dc2d656b3fd730b0075148ba06fb6c79ed913a.log]
2021-02-03T12:24:25.014Z INFO log/input.go:157 Configured paths: [/var/log/containers/*39501043955e8979f76252c7fdf86a610842a8295f49e71499abe0ea6cb39812.log]
2021-02-03T12:24:25.015Z INFO log/input.go:157 Configured paths: [/var/log/containers/*39501043955e8979f76252c7fdf86a610842a8295f49e71499abe0ea6cb39812.log]
2021-02-03T12:24:25.015Z INFO log/input.go:157 Configured paths: [/var/log/containers/*118e8c8674db6b555d81a87ee1306925bc4e60dd60ac2b2320dba2d34b777e15.log]
2021-02-03T12:24:25.015Z INFO log/input.go:157 Configured paths: [/var/log/containers/*118e8c8674db6b555d81a87ee1306925bc4e60dd60ac2b2320dba2d34b777e15.log]
2021-02-03T12:24:25.215Z INFO log/input.go:157 Configured paths: [/var/log/containers/*09083c52807b7457e116845f4a9709626f9ad496bd0113269b21d4d4da3fa026.log]
2021-02-03T12:24:25.216Z INFO log/input.go:157 Configured paths: [/var/log/containers/*09083c52807b7457e116845f4a9709626f9ad496bd0113269b21d4d4da3fa026.log]
2021-02-03T12:24:27.898Z INFO [add_cloud_metadata] add_cloud_metadata/add_cloud_metadata.go:89 add_cloud_metadata: hosting provider type not detected.
2021-02-03T12:24:54.912Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cgroup":{"cpu":{"id":"docker-09083c52807b7457e116845f4a9709626f9ad496bd0113269b21d4d4da3fa026.scope"},"cpuacct":{"id":"docker-09083c52807b7457e116845f4a9709626f9ad496bd0113269b21d4d4da3fa026.scope"},"memory":{"id":"docker-09083c52807b7457e116845f4a9709626f9ad496bd0113269b21d4d4da3fa026.scope"}},"cpu":{"system":{"ticks":50,"time":{"ms":52}},"total":{"ticks":150,"time":{"ms":160},"value":150},"user":{"ticks":100,"time":{"ms":108}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"a8e8c14d-b062-4d4c-9d31-6025d39af801","uptime":{"ms":30087}},"memstats":{"gc_next":19111904,"memory_alloc":14552312,"memory_total":49846672,"rss":47910912},"runtime":{"goroutines":101}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":8,"starts":8}},"output":{"type":"elasticsearch"},"pipeline":{"clients":8,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":2},"load":{"1":0.1,"15":0.34,"5":0.26,"norm":{"1":0.05,"15":0.17,"5":0.13}}}}}}

I have deployed filebeat using https://raw.githubusercontent.com/elastic/beats/7.10/deploy/kubernetes/filebeat-kubernetes.yaml
And here only image version changed to 7.10.0 and elasticsearch host & port info changed as follow -

output.elasticsearch:
  hosts: ["http://10.77.XX.XX:30339"]
  #hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
  username: ${ELASTICSEARCH_USERNAME}
  password: ${ELASTICSEARCH_PASSWORD}

env:
- name: ELASTICSEARCH_HOST
value: "10.77.xx.xx"
#value: elasticsearch
- name: ELASTICSEARCH_PORT
value: "30339"
#value: "9200"
- name: ELASTICSEARCH_USERNAME
value: elastic
- name: ELASTICSEARCH_PASSWORD
value: changeme

It's also observed that deployed apps log info not reflected in filebeat pod log, only metric info showing. I am able to curl elasticsearch from filebeat container.

curl http://xx.xx.xx.xx:30339
{
"name" : "elasticsearch-0",
"cluster_name" : "docker-cluster",
"cluster_uuid" : "z06Zn4p-T3CzUk6hoSiDLg",
"version" : {
"number" : "7.10.0",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "51e9d6f22758d0374a0f3f5c6e8f3a7997850f96",
"build_date" : "2020-11-09T21:30:33.964949Z",
"build_snapshot" : false,
"lucene_version" : "8.7.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}

Please help me to identify the issue.

shaunak · February 3, 2021, 5:55pm

For debugging purposes, lets take Elasticsearch and the network out of the equation. In the Filebeat configuration, could you add enabled: false under output.elasticsearch: and add output.console.enabled: true? Then re-run Filebeat and see if you see any events on STDOUT. That will help us narrow down where the problem is happening.

Shaunak

nirmal_elastic · February 3, 2021, 8:26pm

Thanks Shaunak for reply, pls find below the logs shown on console -

kubectl -n elasticsearch logs filebeat-5mfbc
2021-02-03T20:05:34.780Z	INFO	instance/beat.go:645	Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2021-02-03T20:05:34.780Z	INFO	instance/beat.go:653	Beat ID: 0d19ee72-b91b-45d6-a68f-5ca385adac31
2021-02-03T20:05:34.784Z	INFO	[seccomp]	seccomp/seccomp.go:124	Syscall filter successfully installed
2021-02-03T20:05:34.784Z	INFO	[beat]	instance/beat.go:981	Beat info	{"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "0d19ee72-b91b-45d6-a68f-5ca385adac31"}}}
2021-02-03T20:05:34.784Z	INFO	[beat]	instance/beat.go:990	Build info	{"system_info": {"build": {"commit": "1428d58cf2ed945441fb2ed03961cafa9e4ad3eb", "libbeat": "7.10.0", "time": "2020-11-09T19:57:04.000Z", "version": "7.10.0"}}}
2021-02-03T20:05:34.784Z	INFO	[beat]	instance/beat.go:993	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.14.7"}}}
2021-02-03T20:05:34.786Z	INFO	[beat]	instance/beat.go:997	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-07-16T11:24:22Z","containerized":true,"name":"dcpprsapp014","ip":["fe80::30e2:afff:feb1:cd98/64","127.0.0.1/8","::1/128","10.77.xx.xx/24","fe80::250:56ff:fe8e:9d1b/64","172.17.0.1/16","fe80::42:5dff:fe2c:fd50/64","192.168.205.128/32","fe80::a48f:dcff:fe3c:f12f/64","fe80::b4ae:c4ff:fe70:96f2/64","fe80::6ca0:e4ff:fe4e:1b4c/64","fe80::345d:7bff:fe65:bfef/64"],"kernel_version":"3.10.0-1062.1.2.el7.x86_64","mac":["32:e2:af:b1:cd:98","00:50:56:8e:9d:1b","02:42:5d:2c:fd:50","a6:8f:dc:3c:f1:2f","b6:ae:c4:70:96:f2","6e:a0:e4:4e:1b:4c","36:5d:7b:65:bf:ef"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":8,"patch":2003,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
2021-02-03T20:05:34.786Z	INFO	[beat]	instance/beat.go:1026	Process info	{"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2021-02-03T20:05:33.710Z"}}}
2021-02-03T20:05:34.786Z	INFO	instance/beat.go:299	Setup Beat: filebeat; Version: 7.10.0
2021-02-03T20:05:34.786Z	INFO	[publisher]	pipeline/module.go:113	Beat name: dcpprsapp014
2021-02-03T20:05:34.787Z	WARN	beater/filebeat.go:178	Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2021-02-03T20:05:34.787Z	INFO	instance/beat.go:455	filebeat start running.
2021-02-03T20:05:34.788Z	INFO	memlog/store.go:119	Loading data file of '/usr/share/filebeat/data/registry/filebeat' succeeded. Active transaction id=0
2021-02-03T20:05:34.788Z	INFO	memlog/store.go:124	Finished loading transaction log file for '/usr/share/filebeat/data/registry/filebeat'. Active transaction id=0
2021-02-03T20:05:34.788Z	WARN	beater/filebeat.go:381	Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2021-02-03T20:05:34.788Z	INFO	[registrar]	registrar/registrar.go:109	States Loaded from registrar: 0
2021-02-03T20:05:34.788Z	INFO	[crawler]	beater/crawler.go:71	Loading Inputs: 0
2021-02-03T20:05:34.788Z	INFO	[crawler]	beater/crawler.go:108	Loading and starting Inputs completed. Enabled inputs: 0
2021-02-03T20:05:34.790Z	INFO	[autodiscover.pod]	kubernetes/util.go:99	kubernetes: Using node dcpprsapp014 provided in the config
2021-02-03T20:05:34.790Z	INFO	[autodiscover]	autodiscover/autodiscover.go:113	Starting autodiscover manager
2021-02-03T20:05:34.790Z	INFO	[monitoring]	log/log.go:118	Starting metrics logging every 30s
2021-02-03T20:05:34.891Z	INFO	log/input.go:157	Configured paths: [/var/log/containers/*4a516c5b56d31e22e04920e84eea7b0ab87cafa1065bde22ce1d62298c7cf14f.log]
2021-02-03T20:05:34.892Z	INFO	log/input.go:157	Configured paths: [/var/log/containers/*4a516c5b56d31e22e04920e84eea7b0ab87cafa1065bde22ce1d62298c7cf14f.log]
2021-02-03T20:05:34.892Z	INFO	log/input.go:157	Configured paths: [/var/log/containers/*acf60d277dc422487948026ea4784c227ff992bc444229040a2ea501ae10bd44.log]
2021-02-03T20:05:34.892Z	INFO	log/input.go:157	Configured paths: [/var/log/containers/*acf60d277dc422487948026ea4784c227ff992bc444229040a2ea501ae10bd44.log]
2021-02-03T20:05:34.892Z	INFO	log/input.go:157	Configured paths: [/var/log/containers/*3155747d3897d0167f863c2bb6ba2b58988e3e2021d1211f80f23eeefef0f257.log]
2021-02-03T20:05:34.893Z	INFO	log/input.go:157	Configured paths: [/var/log/containers/*3155747d3897d0167f863c2bb6ba2b58988e3e2021d1211f80f23eeefef0f257.log]
2021-02-03T20:05:34.893Z	INFO	log/input.go:157	Configured paths: [/var/log/containers/*130d718d9830a839927a9a8452f32a1072d302137e1a102c75f6a64ed38ac876.log]
2021-02-03T20:05:34.893Z	INFO	log/input.go:157	Configured paths: [/var/log/containers/*130d718d9830a839927a9a8452f32a1072d302137e1a102c75f6a64ed38ac876.log]
2021-02-03T20:05:34.893Z	INFO	log/input.go:157	Configured paths: [/var/log/containers/*e5f25098747ac271d2e3a73aac1a2814226fea0d9104f21d92c836cd293c7730.log]
2021-02-03T20:05:34.894Z	INFO	log/input.go:157	Configured paths: [/var/log/containers/*e5f25098747ac271d2e3a73aac1a2814226fea0d9104f21d92c836cd293c7730.log]
2021-02-03T20:05:34.894Z	INFO	log/input.go:157	Configured paths: [/var/log/containers/*3e467fd5c6a3972b9fe13acc62f68f25fdea479652bfab5509b55641fef9d753.log]
2021-02-03T20:05:34.894Z	INFO	log/input.go:157	Configured paths: [/var/log/containers/*3e467fd5c6a3972b9fe13acc62f68f25fdea479652bfab5509b55641fef9d753.log]
2021-02-03T20:05:34.898Z	INFO	log/input.go:157	Configured paths: [/var/log/containers/*85cdd5fc4d60454649707ef5a45b8f5bb99ba7eca691ae70720d05d253188503.log]
2021-02-03T20:05:34.898Z	INFO	log/input.go:157	Configured paths: [/var/log/containers/*85cdd5fc4d60454649707ef5a45b8f5bb99ba7eca691ae70720d05d253188503.log]
2021-02-03T20:05:35.132Z	INFO	log/input.go:157	Configured paths: [/var/log/containers/*92dd69246815b05b9bc2a61e74ddc9717886eace1ac590416756a12433da6e29.log]
2021-02-03T20:05:35.133Z	INFO	log/input.go:157	Configured paths: [/var/log/containers/*92dd69246815b05b9bc2a61e74ddc9717886eace1ac590416756a12433da6e29.log]
2021-02-03T20:05:37.782Z	INFO	[add_cloud_metadata]	add_cloud_metadata/add_cloud_metadata.go:89	add_cloud_metadata: hosting provider type not detected.
2021-02-03T20:06:04.794Z	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cgroup":{"cpu":{"id":"docker-92dd69246815b05b9bc2a61e74ddc9717886eace1ac590416756a12433da6e29.scope"},"cpuacct":{"id":"docker-92dd69246815b05b9bc2a61e74ddc9717886eace1ac590416756a12433da6e29.scope"},"memory":{"id":"docker-92dd69246815b05b9bc2a61e74ddc9717886eace1ac590416756a12433da6e29.scope"}},"cpu":{"system":{"ticks":50,"time":{"ms":56}},"total":{"ticks":180,"time":{"ms":187},"value":180},"user":{"ticks":130,"time":{"ms":131}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"b6df6ede-f753-4a9b-883b-dbbdca2695b0","uptime":{"ms":30078}},"memstats":{"gc_next":20819808,"memory_alloc":10801264,"memory_total":49704424,"rss":48422912},"runtime":{"goroutines":101}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":8,"starts":8}},"output":{"type":"console"},"pipeline":{"clients":8,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":2},"load":{"1":0,"15":0.08,"5":0.03,"norm":{"1":0,"15":0.04,"5":0.015}}}}}}
2021-02-03T20:06:34.794Z	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":60,"time":{"ms":11}},"total":{"ticks":190,"time":{"ms":14},"value":190},"user":{"ticks":130,"time":{"ms":3}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"b6df6ede-f753-4a9b-883b-dbbdca2695b0","uptime":{"ms":60078}},"memstats":{"gc_next":20819808,"memory_alloc":11427928,"memory_total":50331088},"runtime":{"goroutines":101}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":8}},"pipeline":{"clients":8,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.2,"15":0.09,"5":0.08,"norm":{"1":0.1,"15":0.045,"5":0.04}}}}}}
2021-02-03T20:07:04.794Z	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":70,"time":{"ms":5}},"total":{"ticks":210,"time":{"ms":12},"value":210},"user":{"ticks":140,"time":{"ms":7}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"b6df6ede-f753-4a9b-883b-dbbdca2695b0","uptime":{"ms":90082}},"memstats":{"gc_next":20819808,"memory_alloc":12029320,"memory_total":50932480},"runtime":{"goroutines":101}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":8}},"pipeline":{"clients":8,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.12,"15":0.09,"5":0.07,"norm":{"1":0.06,"15":0.045,"5":0.035}}}}}}

  I am also attaching the list of deployment that I have made on 3 node K8s cluster-
   a. Elasticsearch - single node, running on master node of K8s
   b. Kibana -   single instance running on master node.
   c. Filebeat- running on all nodes, including master.

 <pre>kubectl -n elasticsearch get all 
NAME                         READY   STATUS    RESTARTS   AGE
pod/elasticsearch-0          1/1     Running   0          6d6h
pod/filebeat-5mfbc           1/1     Running   0          6m34s
pod/filebeat-9nmc6           1/1     Running   0          6m34s
pod/filebeat-tpbzl           1/1     Running   0          6m34s
pod/kibana-b5967c66d-46nt8   1/1     Running   0          6d6h

NAME                    TYPE       CLUSTER-IP        EXTERNAL-IP   PORT(S)          AGE
service/elasticsearch   NodePort   192.168.210.116   &lt;none&gt;        9200:30339/TCP   6d6h
service/kibana          NodePort   192.168.144.0     &lt;none&gt;        5601:30340/TCP   6d6h

NAME                      DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/filebeat   3         3         3       3            3           &lt;none&gt;          6m34s

NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kibana   1/1     1            1           6d6h

NAME                               DESIRED   CURRENT   READY   AGE
replicaset.apps/kibana-b5967c66d   1         1         1       6d6h

NAME                             READY   AGE
statefulset.apps/elasticsearch   1/1     6d6h
</pre>

My expectation is that filebeat will collect the app logs from cluster & I should be able to view/discover it using Kibana. Same is working for metricbeat.

shaunak · February 3, 2021, 9:55pm

Hmm, looking at the logs, it doesn't look like any events are being harvested by Filebeat. I do see a number of containers being discovered by the input though.

@ChrsMark any ideas what might be going on here?

Thanks,

Shaunak

nirmal_elastic · February 4, 2021, 7:05pm

Hi Shaunak,
Is there way to debug it further?
I mean, by being inside filebeat container -
a. message can be pushed/published to elasticsearch node.
b. any command or api call that can be used to harvest log on that node.

ChrsMark · February 5, 2021, 1:12pm

Hi!

For some reason container input seems to not be able to able to collect from logfiles but I'm not sure why this might happen.

Could you please try to replace container input with autodiscover?

At https://raw.githubusercontent.com/elastic/beats/7.10/deploy/kubernetes/filebeat-kubernetes.yaml

Uncomment the section:

   #filebeat.autodiscover:
    #  providers:
    #    - type: kubernetes
    #      node: ${NODE_NAME}
    #      hints.enabled: true
    #      hints.default_config:
    #        type: container
    #        paths:
    #          - /var/log/containers/*${data.kubernetes.container.id}.log

and remove:

    filebeat.inputs:
    - type: container
      paths:
        - /var/log/containers/*.log
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"

Let's see if that works.

nirmal_elastic · February 5, 2021, 7:41pm

Hi Chris!!!

I tried both before putting it in forum, non is working and shared log is basically of  
"filebeat.autodiscover" mode only.

However, I would like to share following points-
  1. My K8s cluster is running on RHEL 7.
  2. No log available at path  "/var/log/containers/"
  3. EFK stack is running on separate namespace  "elasticsearch".
  4. My apps are running on "default" namespace and logs are being redirected to 
      stdout.  
  5. Same setup working fine with metricbeat & able to discover it using Kibana.
      https://raw.githubusercontent.com/elastic/beats/7.10/deploy/kubernetes/metricbeat-kubernetes.yaml

Thanks,
Nirmal

nirmal_elastic · February 5, 2021, 7:54pm

One more point I would like to highlight, I don't know whether helpful or not?
If we check enabled module inside any filebeat container then it's found that
none of modules are in enabled state.

[root@dcpprsapp014 filebeat]# ./filebeat modules list
Enabled:

Disabled:
activemq
apache
....
....

ChrsMark · February 8, 2021, 10:08am

Hey!

Most probably the issue is that /var/log/containers/ is empty.
Please check if inside Filebeat container this dir has any log files. If not check why the Daemonset manifest cannot properly mount the respective directory of the host. Maybe it is because permissions?

Also for autodiscovery you don't need any modules enabled so it's normal that you dont see any to be enabled.

nirmal_elastic · February 8, 2021, 2:28pm

Thanks!!! Chris for looking into the problem. I changed the directory permission from 755 to 777 but seems no effect.

ls -ltr /var/lib/filebeat-data/. ——— changed from 755 to 777
drwxrwxrwx. 3 root root 60 Feb 8 19:33 filebeat-data

ls -ltr /var/lib/filebeat-data/
total 4
-rwxrwxrwx. 1 root root 48 Jan 20 20:34 meta.json
drwxrwxrwx. 3 root root 22 Jan 20 20:34 registry
-rw-------. 1 root root 0 Feb 8 19:33 filebeat.lock

Application and pod level directory permission changed from 755 to 777. Also, /var/log directory properly mounted in pod.

drwxrwxrwx. 2 root root 6 Aug 27 17:20 containers

ls -ltr /var/log/containers/
total 0

Default working directory in interactive mode execution of pod is “/usr/share/filebeat”

Volume mounting part of yaml file -

    volumeMounts:
    - name: config
      mountPath: /etc/filebeat.yml
      readOnly: true
      subPath: filebeat.yml
    - name: data
      mountPath: /usr/share/filebeat/data
    - name: varlibdockercontainers
      mountPath: /var/lib/docker/containers
      #readOnly: true
    - name: varlog
      mountPath: /var/log
      #readOnly: true
  volumes:
  - name: config
    configMap:
      defaultMode: 0640
      name: filebeat-config
  - name: varlibdockercontainers
    hostPath:
      path: /var/lib/docker/containers
  - name: varlog
    hostPath:
      path: /var/log
  # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
  - name: data
    hostPath:
      # When filebeat runs as non-root user, this directory needs to be writable by group (g+w).
      path: /var/lib/filebeat-data
      type: DirectoryOrCreate

Note- tried by both commenting and uncommenting “readonly” part, seems same log. One of Filbeat pod log info is as follow -

nirmal_elastic · February 8, 2021, 2:30pm

'''
kubectl -n elasticsearch logs filebeat-hncd8

2021-02-08T14:03:28.968Z INFO instance/beat.go:645 Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]

2021-02-08T14:03:28.969Z INFO instance/beat.go:653 Beat ID: 86fdb6c7-bf2b-4d58-ae75-6ca3159b21a0

2021-02-08T14:03:28.976Z INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed

2021-02-08T14:03:28.976Z INFO [beat] instance/beat.go:981 Beat info {"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "86fdb6c7-bf2b-4d58-ae75-6ca3159b21a0"}}}

2021-02-08T14:03:28.976Z INFO [beat] instance/beat.go:990 Build info {"system_info": {"build": {"commit": "1428d58cf2ed945441fb2ed03961cafa9e4ad3eb", "libbeat": "7.10.0", "time": "2020-11-09T19:57:04.000Z", "version": "7.10.0"}}}

2021-02-08T14:03:28.976Z INFO [beat] instance/beat.go:993 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.14.7"}}}

2021-02-08T14:03:28.980Z INFO [beat] instance/beat.go:997 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-07-16T11:30:05Z","containerized":true,"name":"dcpprsapp012","ip":["127.0.0.1/8","::1/128","10.77.XX.XX/24","fe80::250:56ff:fe8e:e99c/64","172.17.0.1/16","fe80::42:2ff:fe8e:5057/64","fe80::3492:2ff:fe8b:3e4c/64","10.32.0.1/12","10.32.0.2/12","10.32.0.3/12","fe80::8849:4dff:fe52:4592/64","fe80::45c:88ff:fe99:669b/64","fe80::482:daff:fef3:d300/64","fe80::1067:24ff:fedf:8d9b/64","fe80::38c9:75ff:fead:7849/64","fe80::a4ea:faff:fef2:b277/64","192.168.65.192/32","fe80::c41:f8ff:fe3d:4687/64","fe80::ec80:56ff:fe68:7107/64","fe80::1ceb:2ff:fe67:b366/64","fe80::24:63ff:fe00:2857/64","fe80::dcdf:6fff:fe82:5015/64"],"kernel_version":"3.10.0-1062.1.2.el7.x86_64","mac":["00:50:56:8e:e9:9c","02:42:02:8e:50:57","36:92:02:8b:3e:4c","8a:49:4d:52:45:92","96:63:e6:dc:bd:13","06:5c:88:99:66:9b","06:82:da:f3:d3:00","12:67:24:df:8d:9b","3a:c9:75:ad:78:49","a6:ea:fa:f2:b2:77","0e:41:f8:3d:46:87","ee:80:56:68:71:07","1e:eb:02:67:b3:66","02:24:63:00:28:57","de:df:6f:82:50:15"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":8,"patch":2003,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}

2021-02-08T14:03:28.981Z INFO [beat] instance/beat.go:1026 Process info {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2021-02-08T14:03:28.050Z"}}}

2021-02-08T14:03:28.981Z INFO instance/beat.go:299 Setup Beat: filebeat; Version: 7.10.0

2021-02-08T14:03:28.981Z INFO [publisher] pipeline/module.go:113 Beat name: dcpprsapp012

2021-02-08T14:03:28.982Z WARN beater/filebeat.go:178 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.

2021-02-08T14:03:28.982Z INFO [monitoring] log/log.go:118 Starting metrics logging every 30s

2021-02-08T14:03:28.983Z INFO instance/beat.go:455 filebeat start running.

2021-02-08T14:03:28.984Z INFO memlog/store.go:119 Loading data file of '/usr/share/filebeat/data/registry/filebeat' succeeded. Active transaction id=0

2021-02-08T14:03:28.984Z INFO memlog/store.go:124 Finished loading transaction log file for '/usr/share/filebeat/data/registry/filebeat'. Active transaction id=0

2021-02-08T14:03:28.984Z WARN beater/filebeat.go:381 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.

2021-02-08T14:03:28.984Z INFO [registrar] registrar/registrar.go:109 States Loaded from registrar: 0

2021-02-08T14:03:28.984Z INFO [crawler] beater/crawler.go:71 Loading Inputs: 0

2021-02-08T14:03:28.984Z INFO [crawler] beater/crawler.go:108 Loading and starting Inputs completed. Enabled inputs: 0

2021-02-08T14:03:28.988Z INFO [autodiscover.pod] kubernetes/util.go:99 kubernetes: Using node dcpprsapp012 provided in the config

2021-02-08T14:03:28.988Z INFO [autodiscover] autodiscover/autodiscover.go:113 Starting autodiscover manager

2021-02-08T14:03:29.089Z INFO log/input.go:157 Configured paths: [/var/log/containers/*f23a3d6a9bf6da7597cf17ae6f608b0a8dd2935af7b4c1232a31af135417f21a.log]

2021-02-08T14:03:29.090Z INFO log/input.go:157 Configured paths: [/var/log/containers/*f23a3d6a9bf6da7597cf17ae6f608b0a8dd2935af7b4c1232a31af135417f21a.log]

2021-02-08T14:03:29.090Z INFO log/input.go:157 Configured paths: [/var/log/containers/*a8143613f9d045315a36ee6de6698b1b973d6451033d363562beb5e4ecb03d44.log]

2021-02-08T14:03:29.090Z INFO log/input.go:157 Configured paths: [/var/log/containers/*ceb488bb0625f2593e5289578066365cc020d756fb6e950bedbe161cf78b2636.log]

2021-02-08T14:03:29.091Z INFO log/input.go:157 Configured paths: [/var/log/containers/*ceb488bb0625f2593e5289578066365cc020d756fb6e950bedbe161cf78b2636.log]

2021-02-08T14:03:29.091Z INFO log/input.go:157 Configured paths: [/var/log/containers/*3c7544c7947b3c1cd1377df42ddc9d0633619166d81b23769afc5983603ff020.log]

2021-02-08T14:03:29.091Z INFO log/input.go:157 Configured paths: [/var/log/containers/*40ee39fb1284c270c7d789d86ca1fc6fb7ea88cad5d1d22bbf0c39ee2a244110.log]

2021-02-08T14:03:29.092Z INFO log/input.go:157 Configured paths: [/var/log/containers/*40ee39fb1284c270c7d789d86ca1fc6fb7ea88cad5d1d22bbf0c39ee2a244110.log]

2021-02-08T14:03:29.092Z INFO log/input.go:157 Configured paths: [/var/log/containers/*dc14d64d12e6b531f87a7fc5c50f8e2208d5c2d86e86893ac87b9ef37f0c4d7c.log]

2021-02-08T14:03:29.093Z INFO log/input.go:157 Configured paths: [/var/log/containers/*dc14d64d12e6b531f87a7fc5c50f8e2208d5c2d86e86893ac87b9ef37f0c4d7c.log]

2021-02-08T14:03:29.093Z INFO log/input.go:157 Configured paths: [/var/log/containers/*c95c5d60c4148706878f1638efaf877dbd8b5a27ec18f260a7c82b8b9d4cbd09.log]

2021-02-08T14:03:29.093Z INFO log/input.go:157 Configured paths: [/var/log/containers/*f7297b1892c0646d2fa2fe679f74e22c86c963c7b853416430efd5ad1638a53f.log]

2021-02-08T14:03:29.094Z INFO log/input.go:157 Configured paths: [/var/log/containers/*f7297b1892c0646d2fa2fe679f74e22c86c963c7b853416430efd5ad1638a53f.log]

2021-02-08T14:03:29.097Z INFO log/input.go:157 Configured paths: [/var/log/containers/*4d83a7bc395f257095fdb408f9ffb11a74f2c7dddf5a100586036da996a95d41.log]

2021-02-08T14:03:29.098Z INFO log/input.go:157 Configured paths: [/var/log/containers/*36507d47b46d8b228527edd09fb0872b8b93d453cdaae42befe9c3d52e8ae94b.log]

2021-02-08T14:03:29.098Z INFO log/input.go:157 Configured paths: [/var/log/containers/*56b2ca5105c5ba0fb2dcef8d4f7e2a178820ca794d4a917eec29bdbebc09f106.log]

2021-02-08T14:03:29.099Z INFO log/input.go:157 Configured paths: [/var/log/containers/*56b2ca5105c5ba0fb2dcef8d4f7e2a178820ca794d4a917eec29bdbebc09f106.log]

2021-02-08T14:03:29.099Z INFO log/input.go:157 Configured paths: [/var/log/containers/*fa601670cf2cea3bcf7ca0bc7a5b582a4689cdbac274e4460ce3b02a03379904.log]

2021-02-08T14:03:29.099Z INFO log/input.go:157 Configured paths: [/var/log/containers/*fa601670cf2cea3bcf7ca0bc7a5b582a4689cdbac274e4460ce3b02a03379904.log]
'''

nirmal_elastic · February 8, 2021, 2:31pm

2021-02-08T14:03:29.100Z INFO log/input.go:157 Configured paths: [/var/log/containers/*6f7cfe8ed024ef2188c0af233f01d45277d1c49f4ea1601763477a1cb8e71124.log]

2021-02-08T14:03:29.918Z INFO log/input.go:157 Configured paths: [/var/log/containers/*ebdd015a6a8d3c56f3921b3a93ee640f9319b33a63204305293bf844c24bde70.log]

2021-02-08T14:03:29.919Z INFO log/input.go:157 Configured paths: [/var/log/containers/*ebdd015a6a8d3c56f3921b3a93ee640f9319b33a63204305293bf844c24bde70.log]

2021-02-08T14:03:31.972Z INFO [add_cloud_metadata] add_cloud_metadata/add_cloud_metadata.go:89 add_cloud_metadata: hosting provider type not detected.

2021-02-08T14:03:58.988Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cgroup":{"cpu":{"id":"docker-ebdd015a6a8d3c56f3921b3a93ee640f9319b33a63204305293bf844c24bde70.scope"},"cpuacct":{"id":"docker-ebdd015a6a8d3c56f3921b3a93ee640f9319b33a63204305293bf844c24bde70.scope"},"memory":{"id":"docker-ebdd015a6a8d3c56f3921b3a93ee640f9319b33a63204305293bf844c24bde70.scope"}},"cpu":{"system":{"ticks":60,"time":{"ms":64}},"total":{"ticks":180,"time":{"ms":189},"value":180},"user":{"ticks":120,"time":{"ms":125}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"0b3a20c0-f60e-4c77-9b68-877ce366f6b3","uptime":{"ms":30090}},"memstats":{"gc_next":21778976,"memory_alloc":11662408,"memory_total":52415672,"rss":50151424},"runtime":{"goroutines":175}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":14,"starts":14}},"output":{"type":"console"},"pipeline":{"clients":14,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":2},"load":{"1":1.07,"15":0.55,"5":0.5,"norm":{"1":0.535,"15":0.275,"5":0.25}}}}}}

2021-02-08T14:04:28.987Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":70,"time":{"ms":11}},"total":{"ticks":190,"time":{"ms":15},"value":190},"user":{"ticks":120,"time":{"ms":4}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"0b3a20c0-f60e-4c77-9b68-877ce366f6b3","uptime":{"ms":60093}},"memstats":{"gc_next":21778976,"memory_alloc":12634640,"memory_total":53387904},"runtime":{"goroutines":175}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":14}},"pipeline":{"clients":14,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":1.06,"15":0.57,"5":0.56,"norm":{"1":0.53,"15":0.285,"5":0.28}}}}}}

2021-02-08T14:04:58.987Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":80,"time":{"ms":13}},"total":{"ticks":210,"time":{"ms":18},"value":210},"user":{"ticks":130,"time":{"ms":5}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"0b3a20c0-f60e-4c77-9b68-877ce366f6b3","uptime":{"ms":90092}},"memstats":{"gc_next":21778976,"memory_alloc":13425648,"memory_total":54178912},"runtime":{"goroutines":175}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":14}},"pipeline":{"clients":14,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.91,"15":0.57,"5":0.57,"norm":{"1":0.455,"15":0.285,"5":0.285}}}}}}

2021-02-08T14:05:28.987Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":90,"time":{"ms":8}},"total":{"ticks":220,"time":{"ms":13},"value":220},"user":{"ticks":130,"time":{"ms":5}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"0b3a20c0-f60e-4c77-9b68-877ce366f6b3","uptime":{"ms":120094}},"memstats":{"gc_next":21778976,"memory_alloc":14078928,"memory_total":54832192},"runtime":{"goroutines":175}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":14}},"pipeline":{"clients":14,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.79,"15":0.57,"5":0.57,"norm":{"1":0.395,"15":0.285,"5":0.285}}}}}}

2021-02-08T14:05:58.986Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":110,"time":{"ms":24}},"total":{"ticks":260,"time":{"ms":41},"value":260},"user":{"ticks":150,"time":{"ms":17}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"0b3a20c0-f60e-4c77-9b68-877ce366f6b3","uptime":{"ms":150092}},"memstats":{"gc_next":20039008,"memory_alloc":10473608,"memory_total":55773416,"rss":-1634304},"runtime":{"goroutines":175}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":14}},"pipeline":{"clients":14,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":1.14,"15":0.6,"5":0.66,"norm":{"1":0.57,"15":0.3,"5":0.33}}}}}}

2021-02-08T14:06:28.988Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":120,"time":{"ms":5}},"total":{"ticks":280,"time":{"ms":12},"value":280},"user":{"ticks":160,"time":{"ms":7}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"0b3a20c0-f60e-4c77-9b68-877ce366f6b3","uptime":{"ms":180089}},"memstats":{"gc_next":20039008,"memory_alloc":11119416,"memory_total":56419224},"runtime":{"goroutines":175}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":14}},"pipeline":{"clients":14,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.76,"15":0.59,"5":0.61,"norm":{"1":0.38,"15":0.295,"5":0.305}}}}}}

ChrsMark · February 8, 2021, 2:39pm

So, does /var/log/containers contain any log files inside Filebeat container?

nirmal_elastic · February 8, 2021, 3:05pm

No, it does not.

ChrsMark · February 9, 2021, 1:58pm

So it is normal that you don't see any logs to be collected since there are no log files to collect from.

This is an issue on k8s level deployment. Not sure what could be the reason of this and why these hostPaths are not mounted properly? Are you the one maintaining the k8s cluster or someone else. If there is cluster admin then I suggest you should mention this problem and try to figure out why hostPaths are not mounted.

Are you able to mount hostPath volumes at all in this cluster? Maybe to try with another minimal example about this specific verification?

nirmal_elastic · February 10, 2021, 12:50pm

Thanks for diagnosing issue.

Actually hostPath mount is proper. But your statement that there should be log info at /var/log/containers/ has helped in finding further.

I have created K8s cluster on RHEL 7 and using docker provided by them, there docker Logging Driver was set as "journald" which was preventing to create a symlinks at path "/var/log/containers". After removing it, filebeat started capturing log. Thanks again!!!

ChrsMark · February 10, 2021, 3:24pm

Nice!

Quite interesting that symlinks were not created. Just for reference you can always set the path to /var/lib/docker/containers/*/*.log.

C.

nirmal_elastic · February 10, 2021, 7:15pm

system · March 10, 2021, 9:15pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.