I am using the official elastic helm charts for file beats. My config looks like:
filebeatConfig:
filebeat.yml: |
setup.kibana.host: "http://kibana-kibana:80"
setup.dashboards.enabled: true
filebeat.modules:
- module: amp
amp-agent:
enabled: true
var.paths: ["/var/log/containers/amp-agent*"]
amp-server:
enabled: true
var.paths: ["/var/log/containers/amp-server*"]
filebeat.inputs:
- type: container
paths:
- /var/log/containers/*.log
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: "/var/log/containers/"
output.elasticsearch:
host: '${NODE_NAME}'
hosts: '${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}'
I can see the ingest pipeline is created in elastic search and it works if I use the simulate api to send an example document through but the logs from my service don't run through the pipeline so they're missing fields.
What am I doing wrong. All the logs are at /var/log/containers/ and I set the paths for the filesets to look for my specific logs.
Start up logs:
2021-01-13T16:57:21.992Z INFO instance/beat.go:645 Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2021-01-13T16:57:21.992Z INFO instance/beat.go:653 Beat ID: d29bf120-57ba-427b-9a17-d17dfb0cfa03
2021-01-13T16:57:21.994Z INFO [api] api/server.go:62 Starting stats endpoint
2021-01-13T16:57:21.994Z INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed
2021-01-13T16:57:21.994Z INFO [beat] instance/beat.go:981 Beat info {"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "d29bf120-57ba-427b-9a17-d17dfb0cfa03"}}}
2021-01-13T16:57:21.994Z INFO [api] api/server.go:64 Metrics endpoint listening on: 127.0.0.1:5066 (configured: localhost)
2021-01-13T16:57:21.994Z INFO [beat] instance/beat.go:990 Build info {"system_info": {"build": {"commit": "1428d58cf2ed945441fb2ed03961cafa9e4ad3eb", "libbeat": "7.10.0", "time": "2020-11-09T19:57:04.000Z", "version": "7.10.0"}}}
2021-01-13T16:57:21.994Z INFO [beat] instance/beat.go:993 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.14.7"}}}
2021-01-13T16:57:21.995Z INFO [beat] instance/beat.go:997 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-12-14T11:01:56Z","containerized":true,"name":"filebeat-filebeat-p8q4h","ip":["127.0.0.1/8","10.100.3.75/24"],"kernel_version":"5.8.17-200.fc32.x86_64","mac":["0a:58:0a:64:03:4b"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":8,"patch":2003,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
2021-01-13T16:57:21.996Z INFO [beat] instance/beat.go:1026 Process info {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2021-01-13T16:57:20.770Z"}}}
2021-01-13T16:57:21.996Z INFO instance/beat.go:299 Setup Beat: filebeat; Version: 7.10.0
2021-01-13T16:57:21.996Z INFO [index-management] idxmgmt/std.go:184 Set output.elasticsearch.index to 'filebeat-7.10.0' as ILM is enabled.
2021-01-13T16:57:21.996Z INFO eslegclient/connection.go:99 elasticsearch url: http://elasticsearch-master:9200
2021-01-13T16:57:21.997Z INFO [publisher] pipeline/module.go:113 Beat name: filebeat-filebeat-p8q4h
2021-01-13T16:57:22.002Z INFO beater/filebeat.go:117 Enabled modules/filesets: elasticsearch (audit, deprecation, gc, server, slowlog), amp (amp-agent, amp-server), ()
2021-01-13T16:57:22.004Z INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2021-01-13T16:57:22.005Z INFO kibana/client.go:119 Kibana url: http://kibana-kibana:80
2021-01-13T16:57:24.924Z INFO kibana/client.go:119 Kibana url: http://kibana-kibana:80
2021-01-13T17:06:05.668Z INFO instance/beat.go:815 Kibana dashboards successfully loaded.
2021-01-13T17:06:05.668Z INFO instance/beat.go:455 filebeat start running.
2021-01-13T17:06:05.669Z INFO memlog/store.go:119 Loading data file of '/usr/share/filebeat/data/registry/filebeat' succeeded. Active transaction id=19008795
2021-01-13T17:06:06.223Z INFO memlog/store.go:124 Finished loading transaction log file for '/usr/share/filebeat/data/registry/filebeat'. Active transaction id=19031284
2021-01-13T17:06:06.224Z INFO [registrar] registrar/registrar.go:109 States Loaded from registrar: 27
2021-01-13T17:06:06.224Z INFO [crawler] beater/crawler.go:71 Loading Inputs: 3
2021-01-13T17:06:06.244Z INFO log/input.go:157 Configured paths: [/var/log/containers/*.log]
2021-01-13T17:06:06.244Z INFO [crawler] beater/crawler.go:141 Starting input (ID: 14289678679706674907)
2021-01-13T17:06:06.246Z INFO log/input.go:157 Configured paths: [/var/log/containers/amp-server*.log]
2021-01-13T17:06:06.247Z INFO [crawler] beater/crawler.go:141 Starting input (ID: 7994039714221460170)
2021-01-13T17:06:06.247Z INFO log/harvester.go:302 Harvester started for file: /var/log/containers/prometheus-adapter-674674886d-v7wsr_kube-system_prometheus-adapter-0bc71154da53687b3e7eacc263e48dfd1e6f3a52d72087cb7250154507563381.log
2021-01-13T17:06:06.247Z INFO log/harvester.go:302 Harvester started for file: /var/log/containers/prometheus-operator-79db59df56-7cwtg_kube-system_prometheus-45dc9b1938c9706868ff15d1402b517f919a83e9ff7242aa81aa209ca1db4af8.log
2021-01-13T17:06:06.248Z INFO log/input.go:157 Configured paths: [/var/log/containers/amp-agent*.log]
2021-01-13T17:06:06.248Z INFO [crawler] beater/crawler.go:141 Starting input (ID: 49937013723102106)
2021-01-13T17:06:06.248Z INFO [crawler] beater/crawler.go:108 Loading and starting Inputs completed. Enabled inputs: 3
2021-01-13T17:06:06.249Z INFO log/harvester.go:302 Harvester started for file: /var/log/containers/amp-nginx-ingress-nginx-controller-64b77469f-7bcrf_default_controller-89265a71746afd69d1022db86a9d341c71b32066c91c18d7e2872bd8f1653d54.log
2021-01-13T17:06:06.249Z INFO log/harvester.go:302 Harvester started for file: /var/log/containers/amp-server-64799cd857-6qbj6_dev_amp-server-72dc002e87305ad9d7731ab2b52afaa498bb7b5c8c57be2c0fe2fe75f695cf3a.log
2021-01-13T17:06:06.279Z INFO log/harvester.go:302 Harvester started for file: /var/log/containers/kibana-kibana-d9b4db457-chfzm_logging_kibana-8ab33ad18242b411ebf3e7d6fa014d8c0bc47a058f78da2063325205e1363d97.log
2021-01-13T17:06:06.279Z INFO log/harvester.go:302 Harvester started for file: /var/log/containers/filebeat-filebeat-7psbd_logging_filebeat-ae92b985bcd3220f19849e4c3344785b45ab2a8a83c0e2074ef9507e8b71eb19.log
2021-01-13T17:06:06.280Z INFO log/harvester.go:302 Harvester started for file: /var/log/containers/filebeat-filebeat-7psbd_logging_init-amp-filebeat-module-3bc08ac5e9ba9f1cb0f87a37a61b6dcd4ab28a5d719a0247075f6cf8a495fcce.log
2021-01-13T17:06:06.280Z INFO log/harvester.go:302 Harvester started for file: /var/log/containers/prometheus-prometheus-prometheus-0_kube-system_prometheus-d171be3f308de0caf9c02faf534d7e41a4a4c971b30585439d3d2c3a7435d88a.log
2021-01-13T17:06:06.286Z INFO add_kubernetes_metadata/kubernetes.go:71 add_kubernetes_metadata: kubernetes env detected, with version: v1.18.2
2021-01-13T17:06:06.287Z INFO [kubernetes] kubernetes/util.go:99 kubernetes: Using node ben-cluster-ws3brrf72uss-node-0 provided in the config {"libbeat.processor": "add_kubernetes_metadata"}
2021-01-13T17:06:07.249Z INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to backoff(elasticsearch(http://elasticsearch-master:9200))
2021-01-13T17:06:07.249Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2021-01-13T17:06:07.249Z INFO [publisher] pipeline/retry.go:223 done
2021-01-13T17:06:07.253Z INFO [esclientleg] eslegclient/connection.go:314 Attempting to connect to Elasticsearch version 7.10.0
2021-01-13T17:06:07.277Z INFO [license] licenser/es_callback.go:51 Elasticsearch license: Basic
2021-01-13T17:06:07.278Z INFO [esclientleg] eslegclient/connection.go:314 Attempting to connect to Elasticsearch version 7.10.0
2021-01-13T17:06:07.301Z INFO [index-management] idxmgmt/std.go:261 Auto ILM enable success.
2021-01-13T17:06:07.303Z INFO [index-management.ilm] ilm/std.go:139 do not generate ilm policy: exists=true, overwrite=false
2021-01-13T17:06:07.303Z INFO [index-management] idxmgmt/std.go:274 ILM policy successfully loaded.
2021-01-13T17:06:07.303Z INFO [index-management] idxmgmt/std.go:407 Set setup.template.name to '{filebeat-7.10.0 {now/d}-000001}' as ILM is enabled.
2021-01-13T17:06:07.303Z INFO [index-management] idxmgmt/std.go:412 Set setup.template.pattern to 'filebeat-7.10.0-*' as ILM is enabled.
2021-01-13T17:06:07.303Z INFO [index-management] idxmgmt/std.go:446 Set settings.index.lifecycle.rollover_alias in template to {filebeat-7.10.0 {now/d}-000001} as ILM is enabled.
2021-01-13T17:06:07.303Z INFO [index-management] idxmgmt/std.go:450 Set settings.index.lifecycle.name in template to {filebeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2021-01-13T17:06:07.306Z INFO template/load.go:97 Template filebeat-7.10.0 already exists and will not be overwritten.
2021-01-13T17:06:07.306Z INFO [index-management] idxmgmt/std.go:298 Loaded index template.
2021-01-13T17:06:07.307Z INFO [index-management] idxmgmt/std.go:309 Write alias successfully generated.
2021-01-13T17:06:07.311Z INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to backoff(elasticsearch(http://elasticsearch-master:9200)) established
To clarify I receive all the logs in Kibana. Just my specific logs located at /var/log/containers/amp-agent* or /var/log/containers/amp-server* aren't processed with my custom module/ingest pipeline. I can see these logs exist on the node. Example log name:
/var/log/containers/amp-agent-85f4bd79bd-f786r_amp-agent-8f54d8a1621f351b6f978f4b757165d7c821e2f81fc1b3eb8f1772c7a98c7ec6.log
Example manifest.yaml:
module_version: 1.0
var:
- name: paths
default:
- /var/log/linguamatics/amp/amp_*.log
ingest_pipeline: ingest/pipeline.json
input: config/log.yml
Example log.yaml:
type: log
paths:
{{ range $i, $path := .paths }}
- {{$path}}
{{ end }}
exclude_files: [".gz$"]
multiline:
pattern: '^\[\s*(i2e|amp_agent)\s*\]'
negate: true
match: after