i have a requirement where i am collecting custom log from a container in a Kuberentes Cluster via filebeat agent. The same filebeat agent also collects containers logs as well.
Currently the logs are dumped to hostPath and we are collecting via filebeat agent.[see the helm value below]. the filebeat daemonset which is deployed is not taking any individual ILM policy configured custom logs. Instead it is been used the general ILM configuration.
The custom logs are parsed using ingest pipeline where i am updating the index name for the custom logging. we have a requirement to store this custom log for more than 90 days where as kuberentes logs shall be stored for only 15 days. When i change the index, there is no ILM policy configured either.
Allows you to add any config files in /usr/share/filebeat
such as filebeat.yml
filebeatConfig:
filebeat.yml: |
name: "{NODE_HOSTNAME}"
filebeat.inputs:
- type: container
enabled: false
paths:
- /var/log/containers/*.log
processors:
- add_kubernetes_metadata:
host: {NODE_HOSTNAME}
matchers:
- logs_path:
logs_path: "/var/log/containers/"
- type: log
enabled: true
paths:
- /var/log/custom-pod-logs/*/customlogs/*/custom.log
pipeline: custom_applicationlog
multiline.type: pattern
multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
multiline.negate: true
multiline.match: after
setup.ilm.enabled: true
setup.template.name: "custom-logs-application-sit"
setup.template.pattern: "custom-logs-application-sit-*"
setup.ilm.rollover_alias: "custom-logs-application-sit"
setup.ilm.pattern: "{now/d}-000001"
setup.ilm.policy_name: "custom-logs-application-sit"
processors:
- add_kubernetes_metadata:
host: ${NODE_HOSTNAME}
- type: log
enabled: true
paths:
- /var/log/custom-pod-logs/*/customlogs/*/audit/*.audit
pipeline: custom_auditlog
multiline.type: pattern
multiline.pattern: '^[0-9]{4}/[0-9]{2}/[0-9]{2}'
multiline.negate: true
multiline.match: after
setup.ilm.enabled: true
setup.template.name: "custom-logs-audit-sit"
setup.template.pattern: "custom-logs-audit-sit-*"
setup.ilm.rollover_alias: "custom-logs-audit-sit"
setup.ilm.pattern: "{now/d}-000001"
setup.ilm.policy_name: "custom-logs-audit-sit"
processors:
- add_kubernetes_metadata:
host: ${NODE_HOSTNAME}
- type: log
enabled: true
paths:
- /var/log/custom-pod-logs/*/customlogs/*/webserver/*.log
pipeline: custom_weblog
processors:
- add_kubernetes_metadata:
host: ${NODE_HOSTNAME}
setup.template.name: "custom-logs-web-sit"
setup.template.pattern: "custom-logs-web-sit-*"
setup.ilm.rollover_alias: "custom-logs-web-sit"
setup.ilm.pattern: "{now/d}-000001"
setup.ilm.policy_name: "custom-logs-web-sit"
setup.template.name: "filebeat-sit-aks"
setup.template.pattern: "filebeat-sit-aks-*"
setup.ilm.enabled: true
setup.ilm.rollover_alias: "filebeat-sit-aks"
setup.ilm.pattern: "{now/d}-000001"
setup.ilm.policy_name: "filebeat-sit-aks"
output.elasticsearch:
host: '${NODE_HOSTNAME}'
hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
username: ${ELASTICSEARCH_USERNAME}
password: ${ELASTICSEARCH_PASSWORD}
Extra environment variables to append to the DaemonSet pod spec.
This will be appended to the current 'env:' key. You can use any of the kubernetes env
syntax here
extraEnvs:
- name: 'NODE_HOSTNAME'
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: 'NODE_HOST_IP'
valueFrom:
fieldRef:
fieldPath: status.hostIP
extraVolumeMounts:
- name: extras
mountPath: /usr/share/extras
readOnly: true
extraVolumes:
- name: extras
emptyDir: {}
extraContainers: ""
- name: dummy-init
image: busybox
command: ['echo', 'hey']
extraInitContainers:
- name: dummy-init
image: busybox
command: ['echo', 'hey']
envFrom:
- configMapRef:
name: elastic-cloud-configmap
Root directory where Filebeat will write data to in order to persist registry data across pod restarts (file position and other metadata).
hostPathRoot: /var/lib
hostNetworking: false
image: "docker.elastic.co/beats/filebeat"
imageTag: "7.9.0"
imagePullPolicy: "IfNotPresent"
imagePullSecrets:
livenessProbe:
exec:
command:
- sh
- -c
- |
#!/usr/bin/env bash -e
curl --fail 127.0.0.1:5066
failureThreshold: 3
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
readinessProbe:
exec:
command:
- sh
- -c
- |
#!/usr/bin/env bash -e
filebeat test output
failureThreshold: 3
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
Whether this chart should self-manage its service account, role, and associated role binding.
managedServiceAccount: true
additionals labels
labels: {}
podAnnotations: {}
iam.amazonaws.com/role: es-cluster
Various pod security context settings. Bear in mind that many of these have an impact on Filebeat functioning properly.
- User that the container will execute as. Typically necessary to run as root (0) in order to properly collect host container logs.
- Whether to execute the Filebeat containers as privileged containers. Typically not necessarily unless running within environments such as OpenShift.
podSecurityContext:
runAsUser: 0
privileged: false
resources:
requests:
cpu: "100m"
memory: "100Mi"
limits:
cpu: "1000m"
memory: "200Mi"
Custom service account override that the pod will use
serviceAccount: ""
Annotations to add to the ServiceAccount that is created if the serviceAccount value isn't set.
serviceAccountAnnotations: {}
eks.amazonaws.com/role-arn: arn:aws:iam::111111111111:role/k8s.clustername.namespe.serviceaccount
A list of secrets and their paths to mount inside the pod
This is useful for mounting certificates for security other sensitive values
secretMounts:
- name: filebeat-certificates
secretName: filebeat-certificates
path: /usr/share/filebeat/certs
How long to wait for Filebeat pods to stop gracefully
terminationGracePeriod: 30
tolerations:
nodeSelector: {}
affinity: {}
This is the PriorityClass settings as defined in
https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass
priorityClassName: ""
updateStrategy: RollingUpdate
Override various naming aspects of this chart
Only edit these if you know what you're doing
nameOverride: ""
fullnameOverride: ""
