# Custom log collection using filebeat with seperate ILM policy for each log

**URL:** <https://discuss.elastic.co/t/custom-log-collection-using-filebeat-with-seperate-ilm-policy-for-each-log/250432>\
**Category:** Elasticsearch\
**Tags:** docker\
**Created:** [September 30, 2020, 1:41am UTC](https://discuss.elastic.co/t/custom-log-collection-using-filebeat-with-seperate-ilm-policy-for-each-log/250432 "2020-09-30T01:41:52Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![naveenkumarsp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/naveenkumarsp/32/76416_2.png) [@naveenkumarsp](https://discuss.elastic.co/u/naveenkumarsp)\
**Post date:** [September 30, 2020, 1:41am UTC](https://discuss.elastic.co/t/custom-log-collection-using-filebeat-with-seperate-ilm-policy-for-each-log/250432/1 "2020-09-30T01:41:52Z")

</div>

i have a requirement where i am collecting custom log from a container in a Kuberentes Cluster via filebeat agent. The same filebeat agent also collects containers logs as well.

Currently the logs are dumped to hostPath and we are collecting via filebeat agent.[see the helm value below]. the filebeat daemonset which is deployed is not taking any individual ILM policy configured custom logs. Instead it is been used the general ILM configuration.  
The custom logs are parsed using ingest pipeline where i am updating the index name for the custom logging. we have a requirement to store this custom log for more than 90 days where as kuberentes logs shall be stored for only 15 days. When i change the index, there is no ILM policy configured either.

* * *

# Allows you to add any config files in /usr/share/filebeat

# such as filebeat.yml

filebeatConfig:  
filebeat.yml: |  
name: "{NODE\_HOSTNAME}" filebeat.inputs: - type: container enabled: false paths: - /var/log/containers/\*.log processors: - add\_kubernetes\_metadata: host: {NODE\_HOSTNAME}  
matchers:  
- logs\_path:  
logs\_path: "/var/log/containers/"

```
- type: log
  enabled: true
  paths:
    - /var/log/custom-pod-logs/*/customlogs/*/custom.log
  pipeline: custom_applicationlog
  multiline.type: pattern
  multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after
  setup.ilm.enabled: true
  setup.template.name: "custom-logs-application-sit"
  setup.template.pattern: "custom-logs-application-sit-*"
  setup.ilm.rollover_alias: "custom-logs-application-sit"
  setup.ilm.pattern: "{now/d}-000001"
  setup.ilm.policy_name: "custom-logs-application-sit"
  processors:
  - add_kubernetes_metadata:
      host: ${NODE_HOSTNAME}

- type: log
  enabled: true
  paths:
    - /var/log/custom-pod-logs/*/customlogs/*/audit/*.audit
  pipeline: custom_auditlog
  multiline.type: pattern
  multiline.pattern: '^[0-9]{4}/[0-9]{2}/[0-9]{2}'
  multiline.negate: true
  multiline.match: after
  setup.ilm.enabled: true
  setup.template.name: "custom-logs-audit-sit"
  setup.template.pattern: "custom-logs-audit-sit-*"
  setup.ilm.rollover_alias: "custom-logs-audit-sit"
  setup.ilm.pattern: "{now/d}-000001"
  setup.ilm.policy_name: "custom-logs-audit-sit"
  processors:
  - add_kubernetes_metadata:
      host: ${NODE_HOSTNAME}

- type: log
  enabled: true
  paths:
    - /var/log/custom-pod-logs/*/customlogs/*/webserver/*.log
  pipeline: custom_weblog
  processors:
  - add_kubernetes_metadata:
      host: ${NODE_HOSTNAME}
  setup.template.name: "custom-logs-web-sit"
  setup.template.pattern: "custom-logs-web-sit-*"
  setup.ilm.rollover_alias: "custom-logs-web-sit"
  setup.ilm.pattern: "{now/d}-000001"
  setup.ilm.policy_name: "custom-logs-web-sit"
setup.template.name: "filebeat-sit-aks"
setup.template.pattern: "filebeat-sit-aks-*"
setup.ilm.enabled: true
setup.ilm.rollover_alias: "filebeat-sit-aks"
setup.ilm.pattern: "{now/d}-000001"
setup.ilm.policy_name: "filebeat-sit-aks"
output.elasticsearch:
  host: '${NODE_HOSTNAME}'
  hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
  username: ${ELASTICSEARCH_USERNAME}
  password: ${ELASTICSEARCH_PASSWORD}

```

# Extra environment variables to append to the DaemonSet pod spec.

# This will be appended to the current 'env:' key. You can use any of the kubernetes env

# syntax here

extraEnvs:  
- name: 'NODE\_HOSTNAME'  
valueFrom:  
fieldRef:  
fieldPath: spec.nodeName  
- name: 'NODE\_HOST\_IP'  
valueFrom:  
fieldRef:  
fieldPath: status.hostIP

extraVolumeMounts:

# - name: extras

# mountPath: /usr/share/extras

# readOnly: true

extraVolumes:

# - name: extras

# emptyDir: {}

extraContainers: ""

# - name: dummy-init

# image: busybox

# command: ['echo', 'hey']

extraInitContainers:

# - name: dummy-init

# image: busybox

# command: ['echo', 'hey']

envFrom:

- configMapRef:  
name: elastic-cloud-configmap

# Root directory where Filebeat will write data to in order to persist registry data across pod restarts (file position and other metadata).

hostPathRoot: /var/lib  
hostNetworking: false  
image: "[docker.elastic.co/beats/filebeat](http://docker.elastic.co/beats/filebeat)"  
imageTag: "7.9.0"  
imagePullPolicy: "IfNotPresent"  
imagePullSecrets:

livenessProbe:  
exec:  
command:  
- sh  
- -c  
- |  
#!/usr/bin/env bash -e  
curl --fail 127.0.0.1:5066  
failureThreshold: 3  
initialDelaySeconds: 10  
periodSeconds: 10  
timeoutSeconds: 5

readinessProbe:  
exec:  
command:  
- sh  
- -c  
- |  
#!/usr/bin/env bash -e  
filebeat test output  
failureThreshold: 3  
initialDelaySeconds: 10  
periodSeconds: 10  
timeoutSeconds: 5

# Whether this chart should self-manage its service account, role, and associated role binding.

managedServiceAccount: true

# additionals labels

labels: {}

podAnnotations: {}

# [iam.amazonaws.com/role:](http://iam.amazonaws.com/role:) es-cluster

# Various pod security context settings. Bear in mind that many of these have an impact on Filebeat functioning properly.

# 

# - User that the container will execute as. Typically necessary to run as root (0) in order to properly collect host container logs.

# - Whether to execute the Filebeat containers as privileged containers. Typically not necessarily unless running within environments such as OpenShift.

podSecurityContext:  
runAsUser: 0  
privileged: false

resources:  
requests:  
cpu: "100m"  
memory: "100Mi"  
limits:  
cpu: "1000m"  
memory: "200Mi"

# Custom service account override that the pod will use

serviceAccount: ""

# Annotations to add to the ServiceAccount that is created if the serviceAccount value isn't set.

serviceAccountAnnotations: {}

# [eks.amazonaws.com/role-arn:](http://eks.amazonaws.com/role-arn:) arn:aws:iam::111111111111:role/k8s.clustername.namespe.serviceaccount

# A list of secrets and their paths to mount inside the pod

# This is useful for mounting certificates for security other sensitive values

secretMounts:

# - name: filebeat-certificates

# secretName: filebeat-certificates

# path: /usr/share/filebeat/certs

# How long to wait for Filebeat pods to stop gracefully

terminationGracePeriod: 30

tolerations:

nodeSelector: {}

affinity: {}

# This is the PriorityClass settings as defined in

# [https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass](https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass)

priorityClassName: ""

updateStrategy: RollingUpdate

# Override various naming aspects of this chart

# Only edit these if you know what you're doing

nameOverride: ""  
fullnameOverride: ""

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 30, 2020, 1:48am UTC](https://discuss.elastic.co/t/custom-log-collection-using-filebeat-with-seperate-ilm-policy-for-each-log/250432/2 "2020-09-30T01:48:03Z")

</div>

Welcome to our community! 😃

Please reformat your code/logs/config using the `</>` button, or markdown style back ticks. It helps to make things easy to read which helps us help you 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 28, 2020, 1:48am UTC](https://discuss.elastic.co/t/custom-log-collection-using-filebeat-with-seperate-ilm-policy-for-each-log/250432/3 "2020-10-28T01:48:09Z")

</div>

Closed on October 28, 2020, 1:48am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.