# How to parse haproxy logs using logstash and filebeat

**URL:** <https://discuss.elastic.co/t/how-to-parse-haproxy-logs-using-logstash-and-filebeat/241487>\
**Category:** Elasticsearch\
**Created:** [July 16, 2020, 2:57pm UTC](https://discuss.elastic.co/t/how-to-parse-haproxy-logs-using-logstash-and-filebeat/241487 "2020-07-16T14:57:14Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kay\_Khan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kay_khan/32/45028_2.png) [@Kay\_Khan](https://discuss.elastic.co/u/Kay_Khan)\
**Post date:** [July 16, 2020, 2:57pm UTC](https://discuss.elastic.co/t/how-to-parse-haproxy-logs-using-logstash-and-filebeat/241487/1 "2020-07-16T14:57:14Z")

</div>

I have setup logstash and filebeat on my ECK stack so in kibana/elasticsearch i can view logs from kubernetes. I followed this article to set this up [https://medium.com/@raphaeldelio/deploy-logstash-and-filebeat-on-kubernetes-with-eck-ssl-and-filebeat-d9f616737390](https://medium.com/@raphaeldelio/deploy-logstash-and-filebeat-on-kubernetes-with-eck-ssl-and-filebeat-d9f616737390)

I have a haproxy service on kubernetes and i want to parse the `message` field into separate fields so it can be visualized and analysed.

Log example:  
[![enter image description here](https://i.stack.imgur.com/xP59w.png)](https://i.stack.imgur.com/xP59w.png)

I've come across filebeat modules (but im not 100% sure if this is what i need) [https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-modules.html](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-modules.html) . Anyway, ive added the "haproxy" module as you can see from my filebeats config below but it didint change the output of the log.

**filebeat-configmap.yml**

```
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  labels:
    app: filebeat
data:
  filebeat.yml: |-
    filebeat.modules:
      - module: haproxy
        log:
          enabled: true
          
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.enabled: true
          hints.default_config:
            type: container
            paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log
    processors:
      - add_cloud_metadata:
      - add_host_metadata:
    output.logstash:
      hosts: '${LOGSTASH_URL}'

```

**logstash-configmap.yml**

```
apiVersion: v1
kind: ConfigMap
metadata:
  name: logstash-configmap
data:
  logstash.yml: |
    http.host: "0.0.0.0"
    path.config: /usr/share/logstash/pipeline
  logstash.conf: |
    # all input will come from filebeat, no local logs
    input {
      beats {
        port => 5044
      }
    }
    filter {
      grok {
        match => { "message" => "%{TIMESTAMP_ISO8601:Date} %{LOGLEVEL:Level} %{INT:ProcessID} --- \[%{DATA:ThreadName}\] %{JAVACLASS:Class} : %{GREEDYDATA:LogMessage}" }
      }
    }
    output {
      elasticsearch {
        index => "logstash-%{[@metadata][beat]}"
        hosts => ["${ES_HOSTS}"]
        user => "${ES_USER}"
        password => "${ES_PASSWORD}"
        cacert => '/etc/logstash/certificates/ca.crt'
      }
    }

```

Apparently this module is suppose to create some haproxy dashboard ion kibana aswell, but it hasen't.

---

<div class="post-metadata">

**Author:** ![Kay\_Khan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kay_khan/32/45028_2.png) [@Kay\_Khan](https://discuss.elastic.co/u/Kay_Khan)\
**Post date:** [July 17, 2020, 12:23pm UTC](https://discuss.elastic.co/t/how-to-parse-haproxy-logs-using-logstash-and-filebeat/241487/2 "2020-07-17T12:23:47Z")

</div>

still struggling with this if someone can point in the right direction

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 14, 2020, 12:23pm UTC](https://discuss.elastic.co/t/how-to-parse-haproxy-logs-using-logstash-and-filebeat/241487/3 "2020-08-14T12:23:48Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.