Hi there,
I'm currently testing out upgrading Filebeat from 6.x to 7.2. I've taken our existing 6.x config and changed a couple of fields:
filebeat.registry_flush > filebeat.registry.flush
filebeat.config.prospectors > filebeat.config.input
when.equals:
fileset.module: "ingress-nginx"
to
when.equals:
event.module: "ingress-nginx"
The rest of the configuration is the same as it was previously.
We have a custom module, ingress-nginx with two configured filesets; access & error. We use Kubernetes autodiscover + hints annotations to tell Filebeat to use this module for a certain set of pods:
co.elastic.logs/fileset.stderr: error
co.elastic.logs/fileset.stdout: access
co.elastic.logs/module: ingress-nginx
This has been working fine on 6.x however after changing to 7.2 it seems as though Filebeat is somehow using the incorrect the filesets, using the error fileset for stdout stream.
Taking a look at the below debug log from filebeat you can see the annotations are set to use the ingress-nginx module and use fileset access for stdout stream and error for stderr stream.
2019-06-27T19:16:57.696Z DEBUG [processors] processing/processors.go:183 Publish event: {
"@timestamp": "2019-06-27T19:16:57.677Z",
"@metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "7.2.0",
"pipeline": "filebeat-7.2.0-ingress-nginx-error-pipeline"
},
"agent": {
"version": "7.2.0",
"type": "filebeat",
"ephemeral_id": "3c57afc9-923e-41cd-8bb2-87bae6772477",
"hostname": "filebeat-testing-rqv7k",
"id": "5c021faf-e67b-4ad2-83e2-63f11650fca8"
},
"stream": "stdout",
"message": "<REDACTED>",
"fileset": {
"name": "error"
},
"input": {
"type": "container"
},
"event": {
"module": "ingress-nginx",
"dataset": "ingress-nginx.error"
},
"kubernetes": {
"namespace": "ingress-nginx",
"replicaset": {
"name": "ingress-nginx-internal-controller-5d567b6b5d"
},
"labels": {
"RESTARTED_ON": "1561489759",
"app": "ingress-nginx-internal",
"pod-template-hash": "5d567b6b5d",
"purpose": "ingress-nginx"
},
"annotations": {
"co_elastic_logs/fileset_stdout": "access",
"co_elastic_logs/module": "ingress-nginx",
"co_elastic_logs/fileset_stderr": "error"
},
"pod": {
"name": "ingress-nginx-internal-controller-5d567b6b5d-h9dbc",
"uid": "be036817-990d-11e9-a3ba-42010aa400aa"
},
"node": {
"name": "gke-delivery-platform-core-20190401-6dbff651-xk0b"
},
"container": {
"name": "master"
}
},
"host": {
"name": "filebeat-testing-rqv7k"
},
"log": {
"offset": 1244891,
"file": {
"path": "/var/lib/docker/containers/f9cdb5502f7405473b4ca152f4cdf1698cb17472955171568efd923c19d1ef25/f9cdb5502f7405473b4ca152f4cdf1698cb17472955171568efd923c19d1ef25-json.log"
}
},
"service": {
"type": "ingress-nginx"
},
"ecs": {
"version": "1.0.0"
},
"cloud": {
"provider": "gcp",
"instance": {
"id": "3801039341841480502",
"name": "gke-delivery-platform-core-20190401-6dbff651-xk0b"
},
"machine": {
"type": "custom-12-16384"
},
"availability_zone": "europe-west4-b",
"project": {
"id": "<REDACTED>"
}
}
}
However for some reason fileset.name and event.dataset are both showing error, and the pipeline is ultimately going to the error ingest (filebeat-7.2.0-ingress-nginx-error-pipeline).
Our Filebeat config is as below:
http:
enabled: true
host: localhost
port: 5066
logging.level: debug
filebeat.registry.flush: 2s
# Overwrite pipelines with new config
filebeat.overwrite_pipelines: true
filebeat.config:
input:
# Mounted `filebeat-prospectors` configmap:
path: ${path.config}/prospectors.d/*.yml
# Reload prospectors configs as they change:
reload.enabled: false
modules:
path: ${path.config}/modules.d/*.yml
# Reload module configs as they change:
reload.enabled: false
filebeat.autodiscover:
providers:
- type: kubernetes
hints.enabled: true
include_annotations: ['at/logging','co.elastic.logs/module','co.elastic.logs/fileset.stderr','co.elastic.logs/fileset.stdout']
processors:
- add_cloud_metadata: ~
- drop_event:
when:
not:
equals:
kubernetes.annotations.at/logging: 'true'
- drop_event:
when:
equals:
kubernetes.container.name: 'istio-proxy'
- drop_event:
when:
equals:
kubernetes.container.name: 'istio-init'
- decode_json_fields:
fields: ["message"]
process_array: false
max_depth: 1
target: ""
overwrite_keys: true
output.elasticsearch:
hosts: ["<REDACTED>"]
protocol: 'https'
username: '<REDACTED>'
password: "${ELASTICSEARCH_PASSWORD}"
index: "kubernetes-%{+yyyy.MM.dd}"
bulk_max_size: 250
indices:
- index: "ingress-nginx-%{+yyyy.MM.dd}"
when.equals:
event.module: "ingress-nginx"
setup.template.enabled: false
setup.template.name: 'custom-beats'
setup.template.pattern: "beats-*"
setup.template.overwrite: true
setup.template.settings:
index:
number_of_shards: 2
number_of_replicas: 1
codec: best_compression
xpack.monitoring.enabled: false
Any pointers would be appreciated. I've looked through the 6 -> 7 upgrade guide and the breaking changes docs but I've not found anything that stands out
This is trying to push to an ES 6 cluster that I know isn't supported (Just testing atm), however, I believe Elasticsearch isn't involved yet at the point the event is being constructed incorrectly.
You can see in the below they almost seem flipped:
cc/ @exekias -- Sorry to @ you, I just wanted to check if https://github.com/elastic/beats/pull/12348 / https://github.com/elastic/beats/pull/12162 / https://github.com/elastic/beats/issues/12346 could be at all related to this?
Thanks!