Filebeat - Force filebeat to read custom log from different pod paths

Elastic Stack Beats
1

Hello,

using Elastic + Filebeat + kibana in kubernetes. I have a lot of products running in cluster (apache kafka, apache druid, postgresql).

Checking filebeat behavior it harvest logs from /var/lib/docker/containers/

But for Apache druid for example I need to parse logs from /opt/data/indexing-logs/

filebeatConfig:
    filebeat.yml: |
filebeat.inputs:
- type: container
  paths:
    - /var/log/containers/*.log
    - /opt/data/indexing-logs/*.log
  processors:
  - add_kubernetes_metadata:
      host: ${NODE_NAME}
      matchers:
      - logs_path:
          logs_path: "/var/log/containers/"

It is not working. Any suggestion please how to force filebeat to read custom logs path for multiples pods?

Thanks.

2

What does the Filebeat log show when it starts up>?

3

Hello @warkolm thanks for your prompt response.

Below the filebeat log (I will paste in 2 parts):

+ filebeat-filebeat-fswsc › filebeat
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.510Z	INFO	instance/beat.go:645	Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.510Z	INFO	instance/beat.go:653	Beat ID: 4b13942d-0f55-45f0-9b75-cde6a4884cf0
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.511Z	INFO	[api]	api/server.go:62	Starting stats endpoint
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.511Z	INFO	[api]	api/server.go:64	Metrics endpoint listening on: 127.0.0.1:5066 (configured: localhost)
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.512Z	INFO	[seccomp]	seccomp/seccomp.go:124	Syscall filter successfully installed
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.512Z	INFO	[beat]	instance/beat.go:981	Beat info	{"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "4b13942d-0f55-45f0-9b75-cde6a4884cf0"}}}
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.512Z	INFO	[beat]	instance/beat.go:990	Build info	{"system_info": {"build": {"commit": "1428d58cf2ed945441fb2ed03961cafa9e4ad3eb", "libbeat": "7.10.0", "time": "2020-11-09T19:57:04.000Z", "version": "7.10.0"}}}
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.512Z	INFO	[beat]	instance/beat.go:993	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":8,"version":"go1.14.7"}}}
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.513Z	INFO	[beat]	instance/beat.go:997	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-12-21T12:45:02Z","containerized":true,"name":"filebeat-filebeat-fswsc","ip":["127.0.0.1/8","172.17.0.15/16"],"kernel_version":"4.19.150","mac":["02:42:ac:11:00:0f"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":8,"patch":2003,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.513Z	INFO	[beat]	instance/beat.go:1026	Process info	{"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-12-22T01:56:14.170Z"}}}
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.513Z	INFO	instance/beat.go:299	Setup Beat: filebeat; Version: 7.10.0
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.513Z	INFO	[index-management]	idxmgmt/std.go:184	Set output.elasticsearch.index to 'filebeat-7.10.0' as ILM is enabled.
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.513Z	INFO	eslegclient/connection.go:99	elasticsearch url: http://elasticsearch-master:9200
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.513Z	INFO	[publisher]	pipeline/module.go:113	Beat name: filebeat-filebeat-fswsc
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.516Z	INFO	[monitoring]	log/log.go:118	Starting metrics logging every 30s
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.517Z	INFO	instance/beat.go:455	filebeat start running.
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.517Z	INFO	memlog/store.go:119	Loading data file of '/usr/share/filebeat/data/registry/filebeat' succeeded. Active transaction id=211771
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.765Z	INFO	memlog/store.go:124	Finished loading transaction log file for '/usr/share/filebeat/data/registry/filebeat'. Active transaction id=224644
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.768Z	INFO	[registrar]	registrar/registrar.go:109	States Loaded from registrar: 20
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.768Z	INFO	[crawler]	beater/crawler.go:71	Loading Inputs: 1
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.771Z	INFO	log/input.go:157	Configured paths: [/var/log/containers/*.log /opt/data/indexing-logs/*.log]
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.771Z	INFO	[crawler]	beater/crawler.go:141	Starting input (ID: 14613132718497692233)
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.771Z	INFO	[crawler]	beater/crawler.go:108	Loading and starting Inputs completed. Enabled inputs: 1
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.778Z	INFO	log/harvester.go:302	Harvester started for file: /var/log/containers/kube-proxy-nr9m5_kube-system_kube-proxy-9d666a5e092fdcfb40516fc49c95cf3cfd7910fe5d30b9e2612ace226b37495d.log
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.778Z	INFO	log/harvester.go:302	Harvester started for file: /var/log/containers/druid-router-5869576478-mh97d_datastore_druid-508db6cba2255d790b4561afd28aec7502201a00d7a2509fbab73f73c81235b0.log
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.779Z	INFO	log/harvester.go:302	Harvester started for file: /var/log/containers/druid-zookeeper-1_datastore_zookeeper-58c62fd621f8078ba3e609df36620914b9599336a18a1216d2a103810129d4a3.log
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.780Z	INFO	log/harvester.go:302	Harvester started for file: /var/log/containers/elasticsearch-master-0_logging_configure-sysctl-82b09509706ce08dcc67b43b357e118a6a21fcc239cffb453e9aa863c604c022.log
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.781Z	INFO	log/harvester.go:302	Harvester started for file: /var/log/containers/kube-apiserver-minikube_kube-system_kube-apiserver-3e7275c4caacaa65243038ba1848c32265ea5388a774a8d99086a845de24da00.log
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.781Z	INFO	log/harvester.go:302	Harvester started for file: /var/log/containers/coredns-f9fd979d6-9kxx6_kube-system_coredns-6c05118bb686b059a1d8661070b45fa0ba77c0b48c9f8977ecc1846cd1d8a977.log
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.781Z	INFO	log/harvester.go:302	Harvester started for file: /var/log/containers/druid-zookeeper-0_datastore_zookeeper-2d14b77bc1e7b6287220f6fd91be98c8a9ffd7ea96dfc722c573518fd189b4fc.log
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.782Z	INFO	log/harvester.go:302	Harvester started for file: /var/log/containers/storage-provisioner_kube-system_storage-provisioner-2da78621a9d1efb784a3cb679b0781605109079e64ed3aff537e457b910412c0.log
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.782Z	INFO	log/harvester.go:302	Harvester started for file: /var/log/containers/storage-provisioner_kube-system_storage-provisioner-05f43e811c4ac54ca8f46d0163b1cf07e104023b4dfb25b99f1a51155a6ec06b.log
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.783Z	INFO	log/harvester.go:302	Harvester started for file: /var/log/containers/druid-router-5869576478-mh97d_datastore_druid-920cb37cf6c0d77895bcdc1860f84ce277b7f3bd8643448fc21efe24a53a16ef.log
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.783Z	INFO	log/harvester.go:302	Harvester started for file: /var/log/containers/druid-zookeeper-2_datastore_zookeeper-f9551c59c4b17d76bc8986ea9f99842a28ef0414830fe6d4a00fabdae35439fd.log
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.783Z	INFO	log/harvester.go:302	Harvester started for file: /var/log/containers/etcd-minikube_kube-system_etcd-9c0fc73710a8f37ba3de63f262f15ecc391b12f6420f4f53c13949be2215ede1.log
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.785Z	INFO	log/harvester.go:302	Harvester started for file: /var/log/containers/druid-historical-0_datastore_druid-017ce8761cf2f24178dc5db1f04607af9f3efdca3b1e3abc0c57f65471edb61d.log
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.787Z	INFO	add_kubernetes_metadata/kubernetes.go:71	add_kubernetes_metadata: kubernetes env detected, with version: v1.19.4
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.787Z	INFO	[kubernetes]	kubernetes/util.go:99	kubernetes: Using node minikube provided in the config	{"libbeat.processor": "add_kubernetes_metadata"}
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.789Z	INFO	log/harvester.go:302	Harvester started for file: /var/log/containers/elasticsearch-master-0_logging_elasticsearch-2bd10832dd27e1ffda2ca75609de3869b4ce6d6b01c6de04976b1a4bf9d8d29d.log
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.789Z	INFO	log/harvester.go:302	Harvester started for file: /var/log/containers/kube-scheduler-minikube_kube-system_kube-scheduler-6862b44898b19a65a39d77f185da970972756e1b4b59b95c74b40c4463f79cc6.log
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.783Z	INFO	log/harvester.go:302	Harvester started for file: /var/log/containers/druid-middle-manager-0_datastore_druid-ef89f83591a49cd434ef4c12ebe9a8e13d6cae58d148b599f883df1a8ec3f258.log
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.789Z	INFO	log/harvester.go:302	Harvester started for file: /var/log/containers/druid-middle-manager-0_datastore_druid-5500a9c23595c43b76d2dcd89b2c7c1c67427063b90a680b2a462c16cac7a4cc.log
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.790Z	INFO	log/harvester.go:302	Harvester started for file: /var/log/containers/druid-postgresql-0_datastore_druid-postgresql-eb902d960578101217034f7b09f61bd4540ffc63d84ec1f1c5e642bd2d7784be.log
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.795Z	INFO	log/harvester.go:302	Harvester started for file: /var/log/containers/elasticsearch-master-1_logging_configure-sysctl-d16b6879cd3505d6336747f88fefaea6bdc4b6c8546d26bc392ffab13617658a.log
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.795Z	INFO	log/harvester.go:302	Harvester started for file: /var/log/containers/elasticsearch-master-1_logging_elasticsearch-fada2683f4560398c52634b7b6fd53184efb6028bddceaa861b7b45be2cdff16.log
4 
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.797Z	INFO	log/harvester.go:302	Harvester started for file: /var/log/containers/filebeat-filebeat-fswsc_logging_filebeat-2f36b282ffb08d07fb7c03d5fd74878f02e3b3a051d1d77105d30cd45816567c.log
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.797Z	INFO	log/harvester.go:302	Harvester started for file: /var/log/containers/kube-apiserver-minikube_kube-system_kube-apiserver-ce41b505af122106cae4988263f2b35a5f457fd997868f687aa40b88098db1a4.log
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.804Z	INFO	log/harvester.go:302	Harvester started for file: /var/log/containers/coredns-f9fd979d6-9kxx6_kube-system_coredns-247e202981695fe6e3d53079b116a42e6cf89a043ade928d54c03cfec76ec888.log
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.797Z	INFO	log/harvester.go:302	Harvester started for file: /var/log/containers/kibana-kibana-79ccccbb6-mt444_logging_kibana-21f32a635b8f62f782228945b379534bfea0d3d27269d1dfdc9e5345e891e0aa.log
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.797Z	INFO	log/harvester.go:302	Harvester started for file: /var/log/containers/druid-zookeeper-0_datastore_zookeeper-9ae38c5fb1e77505172a495d76505c4a6fdeb1a65c0586bd03d51f203d1f6a15.log
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.797Z	INFO	log/harvester.go:302	Harvester started for file: /var/log/containers/kube-controller-manager-minikube_kube-system_kube-controller-manager-5da227d3c695373eb86f154b71c8da5a5a0e1327c9f4a772aef10d97fd11a653.log
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.861Z	INFO	log/harvester.go:302	Harvester started for file: /var/log/containers/druid-coordinator-64698d9cf8-z8wzp_datastore_druid-12ff8a1c4418c4c97fc4e2256857dee7030993328d66215d0756d3f8d4b21971.log
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.861Z	INFO	log/harvester.go:302	Harvester started for file: /var/log/containers/etcd-minikube_kube-system_etcd-814841a4efea93b08ed49508b7baac514506fc906f15f4b26a769f8ee088d058.log
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.863Z	INFO	log/harvester.go:302	Harvester started for file: /var/log/containers/kube-proxy-nr9m5_kube-system_kube-proxy-466d42406a0862b83e2f289a1945c9f68184fafec0e1facb7f223e51e5197b11.log
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.866Z	INFO	log/harvester.go:302	Harvester started for file: /var/log/containers/kube-scheduler-minikube_kube-system_kube-scheduler-9f4ce882bec319ea34a36ffa30bb2695e20813666afd814b01664b708604c758.log
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.867Z	INFO	log/harvester.go:302	Harvester started for file: /var/log/containers/druid-broker-655ddc5bc8-7trzd_datastore_druid-e2bfff37c4b37268d0d7911c03b287f6026fa3971243fed01cedcdf56f18c2ea.log
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.867Z	INFO	log/harvester.go:302	Harvester started for file: /var/log/containers/kube-controller-manager-minikube_kube-system_kube-controller-manager-0b744ee3721e8d2ff9211921aa828f4551a85012dc4ddcc67335d2b55f203a03.log
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:14.868Z	INFO	log/harvester.go:302	Harvester started for file: /var/log/containers/druid-historical-0_datastore_druid-31c0b864d784b33738fe83388157ed4832b26e54e204b7b5e9b04aa80ad40b33.log
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:15.160Z	INFO	[publisher_pipeline_output]	pipeline/output.go:143	Connecting to backoff(elasticsearch(http://elasticsearch-master:9200))
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:15.160Z	INFO	[publisher]	pipeline/retry.go:219	retryer: send unwait signal to consumer
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:15.161Z	INFO	[publisher]	pipeline/retry.go:223	  done
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:15.164Z	INFO	[esclientleg]	eslegclient/connection.go:314	Attempting to connect to Elasticsearch version 7.9.3
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:15.181Z	INFO	[license]	licenser/es_callback.go:51	Elasticsearch license: Basic
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:15.182Z	INFO	[esclientleg]	eslegclient/connection.go:314	Attempting to connect to Elasticsearch version 7.9.3
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:15.260Z	INFO	[index-management]	idxmgmt/std.go:261	Auto ILM enable success.
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:15.360Z	INFO	[index-management.ilm]	ilm/std.go:139	do not generate ilm policy: exists=true, overwrite=false
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:15.360Z	INFO	[index-management]	idxmgmt/std.go:274	ILM policy successfully loaded.
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:15.360Z	INFO	[index-management]	idxmgmt/std.go:407	Set setup.template.name to '{filebeat-7.10.0 {now/d}-000001}' as ILM is enabled.
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:15.360Z	INFO	[index-management]	idxmgmt/std.go:412	Set setup.template.pattern to 'filebeat-7.10.0-*' as ILM is enabled.
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:15.360Z	INFO	[index-management]	idxmgmt/std.go:446	Set settings.index.lifecycle.rollover_alias in template to {filebeat-7.10.0 {now/d}-000001} as ILM is enabled.
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:15.361Z	INFO	[index-management]	idxmgmt/std.go:450	Set settings.index.lifecycle.name in template to {filebeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:15.395Z	INFO	template/load.go:97	Template filebeat-7.10.0 already exists and will not be overwritten.
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:15.396Z	INFO	[index-management]	idxmgmt/std.go:298	Loaded index template.
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:15.460Z	INFO	[index-management]	idxmgmt/std.go:309	Write alias successfully generated.
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:15.461Z	INFO	[publisher_pipeline_output]	pipeline/output.go:151	Connection to backoff(elasticsearch(http://elasticsearch-master:9200)) established
filebeat-filebeat-fswsc filebeat 2020-12-22T01:56:44.520Z	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cgroup":{"cpu":{"id":"docker-2f36b282ffb08d07fb7c03d5fd74878f02e3b3a051d1d77105d30cd45816567c.scope"},"cpuacct":{"id":"docker-2f36b282ffb08d07fb7c03d5fd74878f02e3b3a051d1d77105d30cd45816567c.scope"},"memory":{"id":"docker-2f36b282ffb08d07fb7c03d5fd74878f02e3b3a051d1d77105d30cd45816567c.scope"}},"cpu":{"system":{"ticks":12740,"time":{"ms":12740}},"total":{"ticks":23320,"time":{"ms":23326},"value":23320},"user":{"ticks":10580,"time":{"ms":10586}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":45},"info":{"ephemeral_id":"b1fb0b1d-d709-4a91-b491-57f57aedbbb0","uptime":{"ms":30065}},"memstats":{"gc_next":79509408,"memory_alloc":66704968,"memory_total":1583403912,"rss":158437376},"runtime":{"goroutines":203}},"filebeat":{"events":{"active":4094,"added":75759,"done":71665},"harvester":{"open_files":33,"running":33,"started":33}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":71632,"active":48,"batches":1435,"total":71680},"read":{"bytes":950828},"type":"elasticsearch","write":{"bytes":90448791}},"pipeline":{"clients":1,"events":{"active":4071,"filtered":33,"published":75703,"retry":50,"total":75736},"queue":{"acked":71632}}},"registrar":{"states":{"current":34,"update":71665},"writes":{"success":1437,"total":1437}},"system":{"cpu":{"cores":8},"load":{"1":2.41,"15":2.72,"5":2.66,"norm":{"1":0.3013,"15":0.34,"5":0.3325}}}}}}

=================================================

And here I am inside the druid's pod checking if logs were created:

/opt/data/indexing-logs $ pwd
/opt/data/indexing-logs

/opt/data/indexing-logs $ ls -ltr 
total 52
-rw-r--r--    1 druid    druid          405 Dec 21 23:49 index_parallel_movies_json_iahefkch_2020-12-21T23:49:10.869Z.report.json
-rw-r--r--    1 druid    druid        45503 Dec 21 23:49 index_parallel_movies_json_iahefkch_2020-12-21T23:49:10.869Z.log

Thanks in advance.

5

Here the filebeat deployment:

lient.go:108: [debug] creating 5 resource(s)
NAME: filebeat
LAST DEPLOYED: Tue Dec 22 06:21:17 2020
NAMESPACE: logging
STATUS: deployed
REVISION: 1
TEST SUITE: None
USER-SUPPLIED VALUES:
affinity: {}
dnsConfig: {}
envFrom: []
extraContainers: ""
extraEnvs: []
extraInitContainers: []
extraVolumeMounts: []
extraVolumes: []
filebeatConfig:
  filebeat.yml: |
    filebeat.inputs:
    - type: container
      paths:
        - /var/log/containers/*.log
        - /opt/data/indexing-logs/*.log
      processors:
      - add_kubernetes_metadata:
          host: ${NODE_NAME}
          matchers:
          - logs_path:
              logs_path: "/var/log/containers/"

    output.elasticsearch:
      host: '${NODE_NAME}'
      hosts: '${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}'
fullnameOverride: ""
hostNetworking: false
hostPathRoot: /var/lib
image: docker.elastic.co/beats/filebeat
imagePullPolicy: IfNotPresent
imagePullSecrets: []
imageTag: 7.10.0
labels: {}
livenessProbe:
  exec:
    command:
    - sh
    - -c
    - |
      #!/usr/bin/env bash -e
      curl --fail 127.0.0.1:5066
  failureThreshold: 3
  initialDelaySeconds: 10
  periodSeconds: 10
  timeoutSeconds: 5
managedServiceAccount: true
nameOverride: ""
nodeSelector: {}
podAnnotations: {}
podSecurityContext:
  privileged: false
  runAsUser: 0
priorityClassName: ""
readinessProbe:
  exec:
    command:
    - sh
    - -c
    - |
      #!/usr/bin/env bash -e
      filebeat test output
  failureThreshold: 3
  initialDelaySeconds: 10
  periodSeconds: 10
  timeoutSeconds: 5
resources:
  limits:
    cpu: 1000m
    memory: 200Mi
  requests:
    cpu: 100m
    memory: 100Mi
secretMounts: []
serviceAccount: ""
serviceAccountAnnotations: {}
terminationGracePeriod: 30
tolerations: []
updateStrategy: RollingUpdate

COMPUTED VALUES:
affinity: {}
dnsConfig: {}
envFrom: []
extraContainers: ""
extraEnvs: []
extraInitContainers: []
extraVolumeMounts: []
extraVolumes: []
filebeatConfig:
  filebeat.yml: |
    filebeat.inputs:
    - type: container
      paths:
        - /var/log/containers/*.log
        - /opt/data/indexing-logs/*.log
      processors:
      - add_kubernetes_metadata:
          host: ${NODE_NAME}
          matchers:
          - logs_path:
              logs_path: "/var/log/containers/"

    output.elasticsearch:
      host: '${NODE_NAME}'
      hosts: '${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}'
fullnameOverride: ""
hostNetworking: false
hostPathRoot: /var/lib
image: docker.elastic.co/beats/filebeat
imagePullPolicy: IfNotPresent
imagePullSecrets: []
imageTag: 7.10.0
labels: {}
livenessProbe:
  exec:
    command:
    - sh
    - -c
    - |
      #!/usr/bin/env bash -e
      curl --fail 127.0.0.1:5066
  failureThreshold: 3
  initialDelaySeconds: 10
  periodSeconds: 10
  timeoutSeconds: 5
managedServiceAccount: true
nameOverride: ""
nodeSelector: {}
podAnnotations: {}
podSecurityContext:
  privileged: false
  runAsUser: 0
priorityClassName: ""
readinessProbe:
  exec:
    command:
    - sh
    - -c
    - |
      #!/usr/bin/env bash -e
      filebeat test output
  failureThreshold: 3
  initialDelaySeconds: 10
  periodSeconds: 10
  timeoutSeconds: 5
resources:
  limits:
    cpu: 1000m
    memory: 200Mi
  requests:
    cpu: 100m
    memory: 100Mi
secretMounts: []
serviceAccount: ""
serviceAccountAnnotations: {}
terminationGracePeriod: 30
tolerations: []
updateStrategy: RollingUpdate

HOOKS:
MANIFEST:
---
# Source: filebeat/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat-filebeat
  annotations:
  labels:
    app: "filebeat-filebeat"
    chart: "filebeat-7.10.0"
    heritage: "Helm"
    release: "filebeat"
---
# Source: filebeat/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-filebeat-config
  labels:
    app: "filebeat-filebeat"
    chart: "filebeat-7.10.0"
    heritage: "Helm"
    release: "filebeat"
data:
  filebeat.yml: |
    filebeat.inputs:
    - type: container
      paths:
        - /var/log/containers/*.log
        - /opt/data/indexing-logs/*.log
      processors:
      - add_kubernetes_metadata:
          host: ${NODE_NAME}
          matchers:
          - logs_path:
              logs_path: "/var/log/containers/"
    
    output.elasticsearch:
      host: '${NODE_NAME}'
      hosts: '${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}'
---
# Source: filebeat/templates/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: filebeat-filebeat-cluster-role
  labels:
    app: "filebeat-filebeat"
    chart: "filebeat-7.10.0"
    heritage: "Helm"
    release: "filebeat"
rules:
- apiGroups:
  - ""
  resources:
  - namespaces
  - nodes
  - pods
  verbs:
  - get
  - list
  - watch
---
# Source: filebeat/templates/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: filebeat-filebeat-cluster-role-binding
  labels:
    app: "filebeat-filebeat"
    chart: "filebeat-7.10.0"
    heritage: "Helm"
    release: "filebeat"
roleRef:
  kind: ClusterRole
  name: filebeat-filebeat-cluster-role
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: filebeat-filebeat
  namespace: logging
---
# Source: filebeat/templates/daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat-filebeat
  labels:
    app: "filebeat-filebeat"
    chart: "filebeat-7.10.0"
    heritage: "Helm"
    release: "filebeat"
spec:
  selector:
    matchLabels:
      app: "filebeat-filebeat"
      release: "filebeat"
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      annotations:
        
        configChecksum: 7741465561aeeb452923df0979d7f98d1c80ce19a17a2c5714a942dc9ca22d1
      name: "filebeat-filebeat"
      labels:
        app: "filebeat-filebeat"
        chart: "filebeat-7.10.0"
        heritage: "Helm"
        release: "filebeat"
    spec:
      serviceAccountName: filebeat-filebeat
      terminationGracePeriodSeconds: 30
      volumes:
      - name: filebeat-config
        configMap:
          defaultMode: 0600
          name: filebeat-filebeat-config
      - name: data
        hostPath:
          path: /var/lib/filebeat-filebeat-logging-data
          type: DirectoryOrCreate
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: varlog
        hostPath:
          path: /var/log
      - name: varrundockersock
        hostPath:
          path: /var/run/docker.sock
      containers:
      - name: "filebeat"
        image: "docker.elastic.co/beats/filebeat:7.10.0"
        imagePullPolicy: "IfNotPresent"
        args:
        - "-e"
        - "-E"
        - "http.enabled=true"
        livenessProbe:
          exec:
            command:
            - sh
            - -c
            - |
              #!/usr/bin/env bash -e
              curl --fail 127.0.0.1:5066
          failureThreshold: 3
          initialDelaySeconds: 10
          periodSeconds: 10
          timeoutSeconds: 5
        readinessProbe:
          exec:
            command:
            - sh
            - -c
            - |
              #!/usr/bin/env bash -e
              filebeat test output
          failureThreshold: 3
          initialDelaySeconds: 10
          periodSeconds: 10
          timeoutSeconds: 5
        resources:
          limits:
            cpu: 1000m
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        securityContext:
          privileged: false
          runAsUser: 0
        volumeMounts:
        - name: filebeat-config
          mountPath: /usr/share/filebeat/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: varlog
          mountPath: /var/log
          readOnly: true
        # Necessary when using autodiscovery; avoid mounting it otherwise
        # See: https://www.elastic.co/guide/en/beats/filebeat/7.10/configuration-autodiscover.html
        - name: varrundockersock
          mountPath: /var/run/docker.sock
          readOnly: true
6

It looks like it's recognising that it should be reading that path.

7

Yeah,

it is my issue. I am In this nightmare for more than 2 weeks trying to make filebeat to read logs from /opt/data/indexing-logs/

Tried filebeat.inputs: and filebeat.autodiscover:

No luck so far. If you are using kubernetes and can simulate the issue and find some fix please share with me.

Really appreciate your time trying to help me.

Thanks !

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.