Nginx module in filebeat configuration

Elastic Stack Beats
,
1

Hello,

Following this thread: Filebeat nginx module
I tried to configure my Beat CRD to use the NGINX module as part of the ECK deployment in a Kubernetes cluster.

I compare the logs results in Kibana before and after the NGINX module implantation and I don't see any change.

Maybe I'm configuring the module incorrectly.
Or Am I missing something?

This is the Beat CRD yaml after it was deployed. I removed all the k8s admin fields for clarity.

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: eck-orchestration-app-filebeat
  namespace: eck-operator
status:
  availableNodes: 12
  elasticsearchAssociationStatus: Established
  expectedNodes: 12
  health: green
  observedGeneration: 2
  version: 8.13.4
spec:
  config:
    filebeat:
      autodiscover:
        providers:
          - hints:
              default_config:
                paths:
                  - /var/log/containers/*${data.kubernetes.container.id}.log
                type: container
              enabled: true
            node: ${NODE_NAME}
            templates:
              - condition:
                  equals:
                    app.kubernetes.io/name: ingress-nginx
                config:
                  - access:
                      enabled: true
                      input:
                        containers.ids:
                          - ${data.kubernetes.container.id}
                        paths:
                          - >-
                            /var/log/containers/${data.kubernetes.container.id}/*.log
                        type: container
                    module: nginx
            type: kubernetes
    setup:
      ilm:
        enabled: true
        policy_name: filebeat-logs
      template:
        settings:
          index:
            lifecycle:
              name: filebeat-logs
              rollover_alias: filebeat-logs-copy
  daemonSet:
    podTemplate:
      metadata:
        creationTimestamp: null
      spec:
        automountServiceAccountToken: true
        containers:
          - env:
              - name: NODE_NAME
                valueFrom:
                  fieldRef:
                    fieldPath: spec.nodeName
            name: filebeat
            resources:
              limits:
                cpu: '1'
                memory: 1Gi
              requests:
                cpu: 100m
                memory: 300Mi
            securityContext:
              runAsUser: 0
            volumeMounts:
              - mountPath: /var/log/containers
                name: varlogcontainers
              - mountPath: /var/log/pods
                name: varlogpods
        serviceAccount: filebeat
        tolerations:
          - effect: NoSchedule
            operator: Exists
        volumes:
          - hostPath:
              path: /var/log/containers
            name: varlogcontainers
          - hostPath:
              path: /var/log/pods
            name: varlogpods
    updateStrategy: {}
  elasticsearchRef:
    name: eck-orchestration-app-elasticsearch
  kibanaRef: {}
  monitoring:
    logs: {}
    metrics: {}
  type: filebeat
  version: 8.13.4

Thanks :slight_smile:

1 Like
2

I'm also trying to setup nginx module with beats crd or hints annotations, any news ?

3

Here is my working configuration

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: elastic # fix the name
  namespace: elastic-system # remove this if you don't use namespace
spec:
  type: filebeat
  elasticsearchRef:
    name: elastic # fix the name
  kibanaRef:
    name: elastic # fix the name
  config:
    # filebeat.inputs: # not needed for nginx module
    # - type: container
    #   paths:
    #   - /var/log/containers/*.log
    filebeat.modules:
    - module: nginx
      ingress_controller:
        enabled: true
        var.paths: ["/var/log/containers/ingress-nginx-controller*.log"]
        input:
          symlinks: true
    # logging.level: debug # can be used for further debugging
  daemonSet:
    podTemplate:
      spec:
        priorityClassName: monitoring
        # hostNetwork: true
        # dnsPolicy: ClusterFirstWithHostNet
        securityContext: # to read logs as root and also open /var/lib/elastic-system/elastic/filebeat-data on host
          runAsUser: 0
        containers:
        - name: filebeat
          volumeMounts:
          - name: varlogcontainers
            mountPath: /var/log/containers
          - name: varlogpods
            mountPath: /var/log/pods
          - name: varlibdockercontainers
            mountPath: /var/lib/docker/containers
        volumes:
        - name: varlogcontainers
          hostPath:
            path: /var/log/containers
        - name: varlogpods
          hostPath:
            path: /var/log/pods
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers

Please note that if you are reading logs from /var/log/containers, they are symlinks.

filebeat.modules do not follow symlinks, while filebeat.inputs do.

So, in that case, you should override symlinks for input config inside of filebeat.modules to support /var/log/containers.

Another option could be using files inside /var/log/pods, which are not symlinks. Although, I didn't try that.

EDIT: Or, you could just use type: container like:

    filebeat.modules:
    - module: nginx
      ingress_controller:
        enabled: true
        var.paths: ["/var/log/containers/ingress-nginx-controller*.log"]
        input:
          type: container
1 Like