# What is the correct way to handle logs by httpcode?

**URL:** <https://discuss.elastic.co/t/what-is-the-correct-way-to-handle-logs-by-httpcode/231985>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [May 11, 2020, 11:09am UTC](https://discuss.elastic.co/t/what-is-the-correct-way-to-handle-logs-by-httpcode/231985 "2020-05-11T11:09:32Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![David\_Oceans](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/david_oceans/32/51452_2.png) [@David\_Oceans](https://discuss.elastic.co/u/David_Oceans)\
**Post date:** [May 11, 2020, 11:09am UTC](https://discuss.elastic.co/t/what-is-the-correct-way-to-handle-logs-by-httpcode/231985/1 "2020-05-11T11:09:33Z")

</div>

Hi!

In my kubernetes cluster I receive logs of some applications that have this format (specifically ambassador).

But I receive these logs inside the " **message**" field in my Elastic and I cannot easily filter later the messages that are of type 2xx, 3xx, 4xx, etc.

I understand that the correct way would be to parse the log before sending it, right? but how can I do it having this format?

> ACCESS [2020-05-11T10:45:03.098Z] "GET /scregistry/sc-instances/xxx-contract-xxx-e940-4103-xxx-5012a8db3147 HTTP/2" 404 - 0 182 61 56 "xx.xx.173.127" "okhttp/3.9.1" "xxxx-a4e6-4402-xxx-b1845d7a9f17" "[api-xxx.xxx-test.com](http://api-xxx.xxx-test.com)" "xx.xx.20.156:80"

Can you help me?

Thank you

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [May 11, 2020, 3:02pm UTC](https://discuss.elastic.co/t/what-is-the-correct-way-to-handle-logs-by-httpcode/231985/2 "2020-05-11T15:02:00Z")

</div>

Hey @David_Oceans,

Are you using autodiscover to collect the logs from your cluster? If that is the case you can setup different configurations per container to parse their logs.

For many services there are predefined [modules](https://www.elastic.co/guide/en/beats/filebeat/7.6/filebeat-modules.html) that can be used, there are no module yet for ambassador, but maybe other modules for web servers as the one for apache or nginx are able to parse these logs.  
If not, you can also try to parse it using processors, for example the [dissect](https://www.elastic.co/guide/en/beats/filebeat/7.6/dissect.html) one could be able to parse these logs.  
If filebeat processors are not enough, you can also [use ingest pipelines](https://www.elastic.co/guide/en/beats/filebeat/7.6/configuring-ingest-node.html).

In any case, feel free to [open a new issue](https://github.com/elastic/beats/issues/new?template=feature-request.md) requesting support for ambassador, describing your use case.

Thanks!

---

<div class="post-metadata">

**Author:** ![David\_Oceans](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/david_oceans/32/51452_2.png) [@David\_Oceans](https://discuss.elastic.co/u/David_Oceans)\
**Post date:** [May 13, 2020, 2:11pm UTC](https://discuss.elastic.co/t/what-is-the-correct-way-to-handle-logs-by-httpcode/231985/3 "2020-05-13T14:11:47Z")

</div>

Hi,

I was investigating and try differents ways to parser my nginx application, but didn't works me.

I'm using **[docker.elastic.co/beats/filebeat:7.3.2](http://docker.elastic.co/beats/filebeat:7.3.2)**

This is my filebeat configmap

```auto
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: logging
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          hints.enabled: true
    filebeat.inputs:
    - type: container
      paths:
        - /var/log/containers/*.log
      multiline.pattern: '^[[:space:]]+(at|\.{3})\b|^Caused by:'
      multiline.negate: false
      multiline.match: after
      processors:
        - add_kubernetes_metadata:
            in_cluster: true
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"
    processors:
      - add_cloud_metadata:
      - add_host_metadata:
      - decode_json_fields:
          fields: ["message"]
          target: "test"
          overwrite_keys: true          
      - add_fields:
          target: ''
          fields:
            gkeclustername: xxx-apps-stage

    cloud.id: ${ELASTIC_CLOUD_ID}
    cloud.auth: ${ELASTIC_CLOUD_AUTH}

    setup.ilm.rollover_alias: "${INDEX_NAME_CLUSTER}"

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}

```

This is my deployment configuration (helm template)

```auto
# Source: APP_NAME/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: APP_NAME
  labels:
    app.kubernetes.io/name: APP_NAME
    helm.sh/chart: APP_NAME-0.0.1
    app.kubernetes.io/instance: release-name
    app.kubernetes.io/version: "1.0"
    app.kubernetes.io/managed-by: Tiller
spec:
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: APP_NAME
      app.kubernetes.io/instance: release-name
  template:
    metadata:
      labels:
        app.kubernetes.io/name: APP_NAME
        app.kubernetes.io/instance: release-name
      annotations:
        co.elastic.logs/fileset.stderr: error
        co.elastic.logs/fileset.stdout: access
        co.elastic.logs/module: nginx
        
    spec:
      containers:
        - name: APP_NAME
          image: ":"
          imagePullPolicy: Always
          ports:
            - name: http
              containerPort: 80
              protocol: TCP
          envFrom:
            - configMapRef:
                name: global-config-env
            - configMapRef:
                name: release-name-configmap
            - secretRef:
                name: release-name-secrets
          livenessProbe:
            httpGet:
              path: /status/health
              port: http
            initialDelaySeconds: 90
            periodSeconds: 20
            timeoutSeconds: 5
            successThreshold: 1
            failureThreshold: 5
          readinessProbe:
            httpGet:
              path: /status/health
              port: http
            initialDelaySeconds: 30
            periodSeconds: 20
            timeoutSeconds: 10
            successThreshold: 1
            failureThreshold: 5
          resources:
            requests:
              cpu: 40m
              memory: 128Mi
            
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: cloud.google.com/gke-nodepool
                operator: In
                values:
                - preemptible-node-pool
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app.kubernetes.io/instance: APP_NAME
            topologyKey: kubernetes.io/hostname
        
      tolerations:
        - effect: NoSchedule
          key: preemptible
          operator: Equal
          value: "true"

```

But when I see the logs in kibana I still viewing the message in text plain

```auto
10.31.0.5 - - [13/May/2020:14:03:47 +0000] "GET /status/health HTTP/1.1" 200 7473 "-" "kube-probe/1.14+" "-"

```

I have to do something in kibana to refresh index or something? the filebeat and annotations are correct?

Thank you very much

---

<div class="post-metadata">

**Author:** ![David\_Oceans](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/david_oceans/32/51452_2.png) [@David\_Oceans](https://discuss.elastic.co/u/David_Oceans)\
**Post date:** [May 14, 2020, 2:16pm UTC](https://discuss.elastic.co/t/what-is-the-correct-way-to-handle-logs-by-httpcode/231985/4 "2020-05-14T14:16:43Z")

</div>

New advances.

If I have this configured

```auto
filebeat.autodiscover:

```

But also this

```auto
    filebeat.inputs:
    - type: container
      paths:
        - /var/log/containers/*.log
      multiline.pattern: '^[[:space:]]+(at|\.{3})\b|^Caused by:'
      multiline.negate: false
      multiline.match: after
      processors:
        - add_kubernetes_metadata:
            in_cluster: true
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"

```

In fact when you download the latest version you see a comment

```auto
curl -L -O https://raw.githubusercontent.com/elastic/beats/master/deploy/kubernetes/filebeat-kubernetes.yaml

```

```auto
    filebeat.inputs:
    - type: container
      paths:
        - /var/log/containers/*.log
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"

    # To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
    #filebeat.autodiscover:
    # providers:
    # - type: kubernetes
    # node: ${NODE_NAME}
    # hints.enabled: true
    # hints.default_config:
    # type: container
    # paths:
    # - /var/log/containers/*${data.kubernetes.container.id}.log

```

So, I get the merge, and this should be the right way, no?

```auto
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.enabled: true
          hints.default_config:
            type: container
            paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log
            multiline.pattern: '^[[:space:]]+(at|\.{3})\b|^Caused by:'
            multiline.negate: false
            multiline.match: after

```

Thank you

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [May 14, 2020, 3:16pm UTC](https://discuss.elastic.co/t/what-is-the-correct-way-to-handle-logs-by-httpcode/231985/5 "2020-05-14T15:16:31Z")

</div>

Does you last configuration work?

---

<div class="post-metadata">

**Author:** ![David\_Oceans](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/david_oceans/32/51452_2.png) [@David\_Oceans](https://discuss.elastic.co/u/David_Oceans)\
**Post date:** [May 14, 2020, 3:19pm UTC](https://discuss.elastic.co/t/what-is-the-correct-way-to-handle-logs-by-httpcode/231985/6 "2020-05-14T15:19:56Z")

</div>

Multipart message I'm not sure.

Looks good the configuration with indentation?

```auto
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: logging
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.enabled: true
          templates:
            - condition:
                equals:
                  kubernetes.labels.app: nginx-ingress
              config:
                - module: nginx
                  ingress_controller:
                    enabled: true
                    input:
                      type: container
                      paths:
                        - /var/log/containers/*-${data.kubernetes.container.id}.log
          hints.default_config:
            type: container
            paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log
            multiline.pattern: '^[[:space:]]+(at|\.{3})\b|^Caused by:'
            multiline.negate: false
            multiline.match: after
    processors:
      - add_cloud_metadata:
      - add_host_metadata:
      - decode_json_fields:
          fields: ["message"]
          target: "custom"
          overwrite_keys: true          
      - add_fields:
          target: ''
          fields:
            gkeclustername: xxx-apps-stage

```

Thank you

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [May 15, 2020, 12:06pm UTC](https://discuss.elastic.co/t/what-is-the-correct-way-to-handle-logs-by-httpcode/231985/7 "2020-05-15T12:06:17Z")

</div>

Indentation looks good, yes.

I recommend you to use hints OR templates, but not both things. They work independently and they can create then multiple configurations for the same container. For example with this configuration, a pod with the `app: nginx-ingress` label would have two configurations, one with the nginx module, and another one with the container input. This can lead to unexpected results.

If you use hints, you can add these annotations into the ingress controller pod so its logs are collected with the nginx module:

```auto
  co.elastic.logs/module: nginx
  co.elastic.logs/fileset.stdout: ingress_controller
  co.elastic.logs/fileset.stderr: error

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 12, 2020, 12:06pm UTC](https://discuss.elastic.co/t/what-is-the-correct-way-to-handle-logs-by-httpcode/231985/8 "2020-06-12T12:06:25Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.