wajika
(wajika)
August 27, 2020, 4:00am
1
filebeat.autodiscover:
providers:
- type: kubernetes
node: ${NODE_NAME}
hints.enabled: false
hints.default_config:
type: container
finished: true
paths:
- "/var/log/containers/*.log"
templates:
- condition:
or:
- equals:
kubernetes.namespace: "front"
- equals:
kubernetes.namespace: "back"
config:
- type: container
paths:
- "/var/log/containers/*.log"
setup.template.enabled: false
processors:
- add_cloud_metadata:
- add_host_metadata:
- add_kubernetes_metadata:
in_cluster: true
host: ${NODE_NAME}
default_matchers.enabled: false
matchers:
- logs_path:
logs_path: "/var/log/containers/"
resource_type: "container"
output.elasticsearch:
hosts: ${ELASTICSEARCH_HOST}
I only need the logs under the back and front namespaces.
The above method did not succeed. How should I modify it?
wajika
(wajika)
August 27, 2020, 6:51am
2
In addition, I have a question, why filebeat will inject information into application events.
jsoriano
(Jaime Soriano)
August 27, 2020, 11:23am
3
Hey @wajika ,
In your configuration you are using a path with a wildcard that would match all the containers in the node. So every configuration generated, for every pod, will try to harvest any file. You need to setup autodiscover in a way that it generates an specific configuration for each container. Also, when using autodiscover, you don't need to use add_kubernetes_metadata
, events should be already enriched by the autodiscover provider.
I think that something like this would work for you:
filebeat.autodiscover:
providers:
- type: kubernetes
node: ${NODE_NAME}
templates:
- condition:
or:
- equals:
kubernetes.namespace: "front"
- equals:
kubernetes.namespace: "back"
config:
- type: container
paths:
- "/var/log/containers/*-${data.kubernetes.container.id}.log"
setup.template.enabled: false
processors:
- add_cloud_metadata:
- add_host_metadata:
output.elasticsearch:
hosts: ${ELASTICSEARCH_HOST}
Other option could be to use hints-based autodiscover, that would allow you to enable collection of logs per namespace using annotations in the namespaces themselves. See my comment about that on this topic: Collect logs for specific containers or namespace in Openshift/Kubernetes
wajika
(wajika)
August 28, 2020, 12:31am
4
filebeat.autodiscover:
providers:
- type: kubernetes
node: ${NODE_NAME}
templates:
- condition:
or:
- equals:
kubernetes.namespace: "front"
- equals:
kubernetes.namespace: "back"
config:
- type: container
paths:
- "/var/log/containers/*-${data.kubernetes.container.id}.log"
multiline.pattern: '^[[:space:]]'
multiline.negate: false
multiline.match: after
setup.template.enabled: false
processors:
- add_cloud_metadata:
- add_host_metadata:
output.elasticsearch:
hosts: ${ELASTICSEARCH_HOST}
I tried to use your configuration, but filebeat reported an error.
ERROR [autodiscover] autodiscover/autodiscover.go:209 Auto discover config check failed for config '{
"docker-json": {
"cri_flags": true,
"format": "auto",
"partial": true,
"stream": "all"
},
"multiline": {
"match": "after",
"negate": false,
"pattern": "^[[:space:]]"
},
"symlinks": true,
"type": "container"
}', won't start runner: each input must have at least one path defined
One more thing, if I use ${data.kubernetes.container.id}, can the container.id field be generated?
I need to use a field to contact other beats data (APM and metricbeat) on the kubernetes cluster
wajika
(wajika)
August 28, 2020, 1:08am
5
filebeat.autodiscover:
providers:
- type: kubernetes
hints.enabled: true
add_resource_metadata:
namespace:
enabled: true
If I use this configuration, can filebeat support both pod annotations and node annotations?
wajika
(wajika)
August 28, 2020, 5:26am
6
I feel that using the "co.elastic.logs/enabled:'false'" method to turn off the log collection of the namespace is not optimal. If there are a lot of namespaces, then I must increase them one by one.
jsoriano
(Jaime Soriano)
August 30, 2020, 11:28am
7
I think the error can be ignored in your case, but you can add a condition to ignore events without container ids, so the error doesn't happen.
Try this configuration:
filebeat.autodiscover:
providers:
- type: kubernetes
node: ${NODE_NAME}
templates:
- condition:
and:
- has_fields: ['kubernetes.container.id']
- or:
- equals:
kubernetes.namespace: "front"
- equals:
kubernetes.namespace: "back"
config:
- type: container
paths:
- "/var/log/containers/*-${data.kubernetes.container.id}.log"
setup.template.enabled: false
processors:
- add_cloud_metadata:
- add_host_metadata:
output.elasticsearch:
hosts: ${ELASTICSEARCH_HOST}
jsoriano
(Jaime Soriano)
August 30, 2020, 11:33am
8
Try with this to add node annotations:
add_resource_metadata:
node:
enabled: true
include_annotations:
- "someannotation"
- "someotherannotation"
jsoriano
(Jaime Soriano)
August 30, 2020, 11:35am
9
Take into account that you wouldn't need to change the configuration, you would only need to add co.elastic.logs/enabled:'true'
annotation to the namespaces you want to collect logs from.
wajika
(wajika)
August 31, 2020, 1:07am
10
I added has_ fields: [' kubernetes.container.id '] still reporting errors.
However, if you say that this error is not a big problem, let it be for the time being.
wajika
(wajika)
August 31, 2020, 1:33am
12
Sorry, I didn't understand you.
I also need a field container.id, can you advice me, how to do that?
Thank you for your reply.
jsoriano
(Jaime Soriano)
August 31, 2020, 9:00am
13
What do you mean? Filebeat should be adding the kubernetes.container.id
field to collected logs.
wajika
(wajika)
August 31, 2020, 9:26am
14
Kubernetes matedata all appeared, but not yet container.id.
kubernetes.container.id Where should field be added?
jsoriano
(Jaime Soriano)
August 31, 2020, 9:38am
15
Umm, is this container being stopped at this moment? Looking at the code the only case where it seems possible to have the container name but not its id is when the pod is being stopped: https://github.com/elastic/beats/blob/7fbbdca91b5cdfcb943ff7f7b7312219ae9986c0/libbeat/autodiscover/providers/kubernetes/pod.go#L339
If this is a normal running container this may be a bug. Are you missing the kubernetes.container.id
in all events?
wajika
(wajika)
September 1, 2020, 1:02am
16
I can confirm that all containers have no container.id field.
Not even in the mapping.
wajika
(wajika)
September 3, 2020, 1:04am
17
Is there a solution to this problem?
jsoriano
(Jaime Soriano)
September 3, 2020, 9:23am
18
This seems unexpected to me, could you try with a released version, like 7.9.0?
wajika
(wajika)
September 4, 2020, 12:52am
19
This missing container field also occurs on APM agent (without kubernetes metadata).
Filebeat 7.9.1 will also not be generated container.id.
jsoriano
(Jaime Soriano)
September 4, 2020, 10:48am
20
Hey @wajika ,
I have tried to reproduce this and there seems to be actually some problem on Beats with the container ids. I have opened an issue in Github for further investigation: https://github.com/elastic/beats/issues/20982
Thanks!