What is the correct way to handle logs by httpcode?

Hi!

In my kubernetes cluster I receive logs of some applications that have this format (specifically ambassador).

But I receive these logs inside the "message" field in my Elastic and I cannot easily filter later the messages that are of type 2xx, 3xx, 4xx, etc.

I understand that the correct way would be to parse the log before sending it, right? but how can I do it having this format?

ACCESS [2020-05-11T10:45:03.098Z] "GET /scregistry/sc-instances/xxx-contract-xxx-e940-4103-xxx-5012a8db3147 HTTP/2" 404 - 0 182 61 56 "xx.xx.173.127" "okhttp/3.9.1" "xxxx-a4e6-4402-xxx-b1845d7a9f17" "api-xxx.xxx-test.com" "xx.xx.20.156:80"

Can you help me?

Thank you

Hey @David_Oceans,

Are you using autodiscover to collect the logs from your cluster? If that is the case you can setup different configurations per container to parse their logs.

For many services there are predefined modules that can be used, there are no module yet for ambassador, but maybe other modules for web servers as the one for apache or nginx are able to parse these logs.
If not, you can also try to parse it using processors, for example the dissect one could be able to parse these logs.
If filebeat processors are not enough, you can also use ingest pipelines.

In any case, feel free to open a new issue requesting support for ambassador, describing your use case.

Thanks!

Hi,

I was investigating and try differents ways to parser my nginx application, but didn't works me.

I'm using docker.elastic.co/beats/filebeat:7.3.2

This is my filebeat configmap

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: logging
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          hints.enabled: true
    filebeat.inputs:
    - type: container
      paths:
        - /var/log/containers/*.log
      multiline.pattern: '^[[:space:]]+(at|\.{3})\b|^Caused by:'
      multiline.negate: false
      multiline.match: after
      processors:
        - add_kubernetes_metadata:
            in_cluster: true
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"
    processors:
      - add_cloud_metadata:
      - add_host_metadata:
      - decode_json_fields:
          fields: ["message"]
          target: "test"
          overwrite_keys: true          
      - add_fields:
          target: ''
          fields:
            gkeclustername: xxx-apps-stage

    cloud.id: ${ELASTIC_CLOUD_ID}
    cloud.auth: ${ELASTIC_CLOUD_AUTH}

    setup.ilm.rollover_alias: "${INDEX_NAME_CLUSTER}"

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}

This is my deployment configuration (helm template)

# Source: APP_NAME/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: APP_NAME
  labels:
    app.kubernetes.io/name: APP_NAME
    helm.sh/chart: APP_NAME-0.0.1
    app.kubernetes.io/instance: release-name
    app.kubernetes.io/version: "1.0"
    app.kubernetes.io/managed-by: Tiller
spec:
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: APP_NAME
      app.kubernetes.io/instance: release-name
  template:
    metadata:
      labels:
        app.kubernetes.io/name: APP_NAME
        app.kubernetes.io/instance: release-name
      annotations:
        co.elastic.logs/fileset.stderr: error
        co.elastic.logs/fileset.stdout: access
        co.elastic.logs/module: nginx
        
    spec:
      containers:
        - name: APP_NAME
          image: ":"
          imagePullPolicy: Always
          ports:
            - name: http
              containerPort: 80
              protocol: TCP
          envFrom:
            - configMapRef:
                name: global-config-env
            - configMapRef:
                name: release-name-configmap
            - secretRef:
                name: release-name-secrets
          livenessProbe:
            httpGet:
              path: /status/health
              port: http
            initialDelaySeconds: 90
            periodSeconds: 20
            timeoutSeconds: 5
            successThreshold: 1
            failureThreshold: 5
          readinessProbe:
            httpGet:
              path: /status/health
              port: http
            initialDelaySeconds: 30
            periodSeconds: 20
            timeoutSeconds: 10
            successThreshold: 1
            failureThreshold: 5
          resources:
            requests:
              cpu: 40m
              memory: 128Mi
            
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: cloud.google.com/gke-nodepool
                operator: In
                values:
                - preemptible-node-pool
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app.kubernetes.io/instance: APP_NAME
            topologyKey: kubernetes.io/hostname
        
      tolerations:
        - effect: NoSchedule
          key: preemptible
          operator: Equal
          value: "true"

But when I see the logs in kibana I still viewing the message in text plain

10.31.0.5 - - [13/May/2020:14:03:47 +0000] "GET /status/health HTTP/1.1" 200 7473 "-" "kube-probe/1.14+" "-"

I have to do something in kibana to refresh index or something? the filebeat and annotations are correct?

Thank you very much

New advances.

If I have this configured

filebeat.autodiscover:

But also this

    filebeat.inputs:
    - type: container
      paths:
        - /var/log/containers/*.log
      multiline.pattern: '^[[:space:]]+(at|\.{3})\b|^Caused by:'
      multiline.negate: false
      multiline.match: after
      processors:
        - add_kubernetes_metadata:
            in_cluster: true
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"

In fact when you download the latest version you see a comment

curl -L -O https://raw.githubusercontent.com/elastic/beats/master/deploy/kubernetes/filebeat-kubernetes.yaml
    filebeat.inputs:
    - type: container
      paths:
        - /var/log/containers/*.log
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"

    # To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
    #filebeat.autodiscover:
    #  providers:
    #    - type: kubernetes
    #      node: ${NODE_NAME}
    #      hints.enabled: true
    #      hints.default_config:
    #        type: container
    #        paths:
    #          - /var/log/containers/*${data.kubernetes.container.id}.log

So, I get the merge, and this should be the right way, no?

    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.enabled: true
          hints.default_config:
            type: container
            paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log
            multiline.pattern: '^[[:space:]]+(at|\.{3})\b|^Caused by:'
            multiline.negate: false
            multiline.match: after

Thank you

Does you last configuration work?

Multipart message I'm not sure.

Looks good the configuration with indentation?

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: logging
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.enabled: true
          templates:
            - condition:
                equals:
                  kubernetes.labels.app: nginx-ingress
              config:
                - module: nginx
                  ingress_controller:
                    enabled: true
                    input:
                      type: container
                      paths:
                        - /var/log/containers/*-${data.kubernetes.container.id}.log
          hints.default_config:
            type: container
            paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log
            multiline.pattern: '^[[:space:]]+(at|\.{3})\b|^Caused by:'
            multiline.negate: false
            multiline.match: after
    processors:
      - add_cloud_metadata:
      - add_host_metadata:
      - decode_json_fields:
          fields: ["message"]
          target: "custom"
          overwrite_keys: true          
      - add_fields:
          target: ''
          fields:
            gkeclustername: xxx-apps-stage

Thank you

Indentation looks good, yes.

I recommend you to use hints OR templates, but not both things. They work independently and they can create then multiple configurations for the same container. For example with this configuration, a pod with the app: nginx-ingress label would have two configurations, one with the nginx module, and another one with the container input. This can lead to unexpected results.

If you use hints, you can add these annotations into the ingress controller pod so its logs are collected with the nginx module:

  co.elastic.logs/module: nginx
  co.elastic.logs/fileset.stdout: ingress_controller
  co.elastic.logs/fileset.stderr: error

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.