Custom log collection using filebeat with seperate ILM policy for each log

i have a requirement where i am collecting custom log from a container in a Kuberentes Cluster via filebeat agent. The same filebeat agent also collects containers logs as well.

Currently the logs are dumped to hostPath and we are collecting via filebeat agent.[see the helm value below]. the filebeat daemonset which is deployed is not taking any individual ILM policy configured custom logs. Instead it is been used the general ILM configuration.
The custom logs are parsed using ingest pipeline where i am updating the index name for the custom logging. we have a requirement to store this custom log for more than 90 days where as kuberentes logs shall be stored for only 15 days. When i change the index, there is no ILM policy configured either.


Allows you to add any config files in /usr/share/filebeat

such as filebeat.yml

filebeatConfig:
filebeat.yml: |
name: "{NODE_HOSTNAME}" filebeat.inputs: - type: container enabled: false paths: - /var/log/containers/*.log processors: - add_kubernetes_metadata: host: {NODE_HOSTNAME}
matchers:
- logs_path:
logs_path: "/var/log/containers/"

- type: log
  enabled: true
  paths:
    - /var/log/custom-pod-logs/*/customlogs/*/custom.log
  pipeline: custom_applicationlog
  multiline.type: pattern
  multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after
  setup.ilm.enabled: true
  setup.template.name: "custom-logs-application-sit"
  setup.template.pattern: "custom-logs-application-sit-*"
  setup.ilm.rollover_alias: "custom-logs-application-sit"
  setup.ilm.pattern: "{now/d}-000001"
  setup.ilm.policy_name: "custom-logs-application-sit"
  processors:
  - add_kubernetes_metadata:
      host: ${NODE_HOSTNAME}

- type: log
  enabled: true
  paths:
    - /var/log/custom-pod-logs/*/customlogs/*/audit/*.audit
  pipeline: custom_auditlog
  multiline.type: pattern
  multiline.pattern: '^[0-9]{4}/[0-9]{2}/[0-9]{2}'
  multiline.negate: true
  multiline.match: after
  setup.ilm.enabled: true
  setup.template.name: "custom-logs-audit-sit"
  setup.template.pattern: "custom-logs-audit-sit-*"
  setup.ilm.rollover_alias: "custom-logs-audit-sit"
  setup.ilm.pattern: "{now/d}-000001"
  setup.ilm.policy_name: "custom-logs-audit-sit"
  processors:
  - add_kubernetes_metadata:
      host: ${NODE_HOSTNAME}

- type: log
  enabled: true
  paths:
    - /var/log/custom-pod-logs/*/customlogs/*/webserver/*.log
  pipeline: custom_weblog
  processors:
  - add_kubernetes_metadata:
      host: ${NODE_HOSTNAME}
  setup.template.name: "custom-logs-web-sit"
  setup.template.pattern: "custom-logs-web-sit-*"
  setup.ilm.rollover_alias: "custom-logs-web-sit"
  setup.ilm.pattern: "{now/d}-000001"
  setup.ilm.policy_name: "custom-logs-web-sit"
setup.template.name: "filebeat-sit-aks"
setup.template.pattern: "filebeat-sit-aks-*"
setup.ilm.enabled: true
setup.ilm.rollover_alias: "filebeat-sit-aks"
setup.ilm.pattern: "{now/d}-000001"
setup.ilm.policy_name: "filebeat-sit-aks"
output.elasticsearch:
  host: '${NODE_HOSTNAME}'
  hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
  username: ${ELASTICSEARCH_USERNAME}
  password: ${ELASTICSEARCH_PASSWORD}

Extra environment variables to append to the DaemonSet pod spec.

This will be appended to the current 'env:' key. You can use any of the kubernetes env

syntax here

extraEnvs:
- name: 'NODE_HOSTNAME'
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: 'NODE_HOST_IP'
valueFrom:
fieldRef:
fieldPath: status.hostIP

extraVolumeMounts:

- name: extras

mountPath: /usr/share/extras

readOnly: true

extraVolumes:

- name: extras

emptyDir: {}

extraContainers: ""

- name: dummy-init

image: busybox

command: ['echo', 'hey']

extraInitContainers:

- name: dummy-init

image: busybox

command: ['echo', 'hey']

envFrom:

  • configMapRef:
    name: elastic-cloud-configmap

Root directory where Filebeat will write data to in order to persist registry data across pod restarts (file position and other metadata).

hostPathRoot: /var/lib
hostNetworking: false
image: "docker.elastic.co/beats/filebeat"
imageTag: "7.9.0"
imagePullPolicy: "IfNotPresent"
imagePullSecrets:

livenessProbe:
exec:
command:
- sh
- -c
- |
#!/usr/bin/env bash -e
curl --fail 127.0.0.1:5066
failureThreshold: 3
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5

readinessProbe:
exec:
command:
- sh
- -c
- |
#!/usr/bin/env bash -e
filebeat test output
failureThreshold: 3
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5

Whether this chart should self-manage its service account, role, and associated role binding.

managedServiceAccount: true

additionals labels

labels: {}

podAnnotations: {}

iam.amazonaws.com/role: es-cluster

Various pod security context settings. Bear in mind that many of these have an impact on Filebeat functioning properly.

- User that the container will execute as. Typically necessary to run as root (0) in order to properly collect host container logs.

- Whether to execute the Filebeat containers as privileged containers. Typically not necessarily unless running within environments such as OpenShift.

podSecurityContext:
runAsUser: 0
privileged: false

resources:
requests:
cpu: "100m"
memory: "100Mi"
limits:
cpu: "1000m"
memory: "200Mi"

Custom service account override that the pod will use

serviceAccount: ""

Annotations to add to the ServiceAccount that is created if the serviceAccount value isn't set.

serviceAccountAnnotations: {}

eks.amazonaws.com/role-arn: arn:aws:iam::111111111111:role/k8s.clustername.namespe.serviceaccount

A list of secrets and their paths to mount inside the pod

This is useful for mounting certificates for security other sensitive values

secretMounts:

- name: filebeat-certificates

secretName: filebeat-certificates

path: /usr/share/filebeat/certs

How long to wait for Filebeat pods to stop gracefully

terminationGracePeriod: 30

tolerations:

nodeSelector: {}

affinity: {}

This is the PriorityClass settings as defined in

https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass

priorityClassName: ""

updateStrategy: RollingUpdate

Override various naming aspects of this chart

Only edit these if you know what you're doing

nameOverride: ""
fullnameOverride: ""

Welcome to our community! :smiley:

Please reformat your code/logs/config using the </> button, or markdown style back ticks. It helps to make things easy to read which helps us help you :slight_smile: