# Custom haproxy module grok pattern failing

**URL:** <https://discuss.elastic.co/t/custom-haproxy-module-grok-pattern-failing/172613>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [March 16, 2019, 1:36am UTC](https://discuss.elastic.co/t/custom-haproxy-module-grok-pattern-failing/172613 "2019-03-16T01:36:31Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![10acc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/10acc/32/96079_2.png) [@10acc](https://discuss.elastic.co/u/10acc)\
**Post date:** [March 16, 2019, 1:36am UTC](https://discuss.elastic.co/t/custom-haproxy-module-grok-pattern-failing/172613/1 "2019-03-16T01:36:31Z")

</div>

I am sending my haproxy logs with filebeat straight to my elasticsearch server.

My haproxy logs are standard HTTP format , just have additional %fp (frontend\_port) at the very end.  
`Mar 16 03:28:45 proksikas haproxy[27622]: 102.92.33.60:58869 [16/Mar/2019:03:27:59.909] Bite Bite/ss72 0/0/0/44/45671 200 396144 - - cD-- 361/361/360/2/0 0/0 "CONNECT gogle:443 HTTP/1.1" 3682` \< 4 digits port at the end.

So I edited `/usr/share/filebeat/module/haproxy/log/ingest/pipeline.json` file accordingly, adding `%{NUMBER:haproxy.fp:long` to the end of every grok pattern.

Interestingly enough in my kibana dashboard I can only see -

```
    haproxy.http.request.raw_request_line
     haproxy.connection_wait_time_ms
    haproxy.fp <<< my custom port

```

But other fields are not exported, and the error is attached to each of the messages in Discovery tab -

`cannot set [address] with parent object of type [java.lang.String] as part of path [source.address]`

**Please shed some light on this, thank you!**

P.s Here's my full haproxy pipeline-

```
{
"description": "Pipeline for parsing HAProxy http, tcp and default logs. Requires the geoip plugin.",
"processors": [
    {
        "grok": {
            "field": "message",
            "patterns": [

                "%{HAPROXY_DATE:haproxy.request_date} %{IPORHOST:haproxy.source} %{PROG:process.name}(?:\\[%{POSINT:process.pid:long}\\])?: %{GREEDYDATA} %{IPORHOST:source.address}:%{POSINT:source.port:long} %{WORD} %{IPORHOST:destination.ip}:%{POSINT:destination.port:long} \\(%{WORD:haproxy.frontend_name}/%{WORD:haproxy.mode}\\) %{NUMBER:haproxy.fp:long}",

                "(%{NOTSPACE:process.name}\\[%{NUMBER:process.pid:long}\\]: )?%{IP:source.address}:%{NUMBER:source.port:long} \\[%{NOTSPACE:haproxy.request_date}\\] %{NOTSPACE:haproxy.frontend_name} %{NOTSPACE:haproxy.backend_name}/%{NOTSPACE:haproxy.server_name} %{NUMBER:haproxy.http.request.time_wait_ms:long}/%{NUMBER:haproxy.total_waiting_time_ms:long}/%{NUMBER:haproxy.connection_wait_time_ms:long}/%{NUMBER:haproxy.http.request.time_wait_without_data_ms:long}/%{NUMBER:temp.duration:long} %{NUMBER:http.response.status_code:long} %{NUMBER:haproxy.bytes_read:long} %{NOTSPACE:haproxy.http.request.captured_cookie} %{NOTSPACE:haproxy.http.response.captured_cookie} %{NOTSPACE:haproxy.termination_state} %{NUMBER:haproxy.connections.active:long}/%{NUMBER:haproxy.connections.frontend:long}/%{NUMBER:haproxy.connections.backend:long}/%{NUMBER:haproxy.connections.server:long}/%{NUMBER:haproxy.connections.retries:long} %{NUMBER:haproxy.server_queue:long}/%{NUMBER:haproxy.backend_queue:long} (\\{%{DATA:haproxy.http.request.captured_headers}\\} \\{%{DATA:haproxy.http.response.captured_headers}\\} |\\{%{DATA}\\} )?\"%{GREEDYDATA:haproxy.http.request.raw_request_line}\" %{NUMBER:haproxy.fp:long}",

                "(%{NOTSPACE:process.name}\\[%{NUMBER:process.pid:long}\\]: )?%{IP:source.address}:%{NUMBER:source.port:long} \\[%{NOTSPACE:haproxy.request_date}\\] %{NOTSPACE:haproxy.frontend_name}/%{NOTSPACE:haproxy.bind_name} %{GREEDYDATA:haproxy.error_message} %{NUMBER:haproxy.fp:long}",

                "%{HAPROXY_DATE} %{IPORHOST:haproxy.source} (%{NOTSPACE:process.name}\\[%{NUMBER:process.pid:long}\\]: )?%{IP:source.address}:%{NUMBER:source.port:long} \\[%{NOTSPACE:haproxy.request_date}\\] %{NOTSPACE:haproxy.frontend_name} %{NOTSPACE:haproxy.backend_name}/%{NOTSPACE:haproxy.server_name} %{NUMBER:haproxy.total_waiting_time_ms:long}/%{NUMBER:haproxy.connection_wait_time_ms:long}/%{NUMBER:temp.duration:long} %{NUMBER:haproxy.bytes_read:long} %{NOTSPACE:haproxy.termination_state} %{NUMBER:haproxy.connections.active:long}/%{NUMBER:haproxy.connections.frontend:long}/%{NUMBER:haproxy.connections.backend:long}/%{NUMBER:haproxy.connections.server:long}/%{NUMBER:haproxy.connections.retries:long} %{NUMBER:haproxy.server_queue:long}/%{NUMBER:haproxy.backend_queue:long} %{NUMBER:haproxy.fp:long}"
            ],
            "ignore_missing": false,
            "pattern_definitions": {
                "HAPROXY_DATE": "(%{MONTHDAY}[/-]%{MONTH}[/-]%{YEAR}:%{HOUR}:%{MINUTE}:%{SECOND})|%{SYSLOGTIMESTAMP}"
            }
        }
    },
    {
        "date": {
            "field": "haproxy.request_date",
            "target_field": "@timestamp",
            "formats": [
                "dd/MMM/yyyy:HH:mm:ss.SSS",
                "MMM dd HH:mm:ss"
            ]
        }
    },
    {
        "remove": {
            "field": "haproxy.request_date"
        }
    },
    {
        "remove": {
            "field": "message"
        }
    },
    {
        "grok": {
            "field": "source.address",
            "ignore_failure": true,
            "patterns": [
                "^%{IP:source.ip}$"
            ]
        }
    },
    {
        "geoip": {
            "field": "source.ip",
            "target_field": "source.geo",
            "ignore_missing": true
        }
    },
    {
        "split": {
            "field": "haproxy.http.request.captured_headers",
            "separator": "\\|",
            "ignore_failure": true
        }
    },
    {
        "split": {
            "field": "haproxy.http.response.captured_headers",
            "separator": "\\|",
            "ignore_failure": true
        }
    },

    {
        "script": {
            "lang": "painless",
            "source": "ctx.event.duration = Math.round(ctx.temp.duration * params.scale)",
            "params": { "scale": 1000000 },
            "if": "ctx.temp?.duration != null"
        }
    },
    {
        "remove": {
            "field": "temp.duration",
            "ignore_missing": true
        }
    },

    {
        "convert": {
            "field": "haproxy.bytes_read",
            "target_field": "http.response.bytes",
            "type": "long",
            "if": "ctx.containsKey('http')"
        }
    }
],
"on_failure": [
    {
        "set": {
            "field": "error.message",
            "value": "{{ _ingest.on_failure_message }}"
        }
    }
]

```

}

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [March 25, 2019, 8:29am UTC](https://discuss.elastic.co/t/custom-haproxy-module-grok-pattern-failing/172613/2 "2019-03-25T08:29:47Z")

</div>

Which filebeat version are you using? I assume your problem is that `source` in Filebeat 6.x is a keyword and only in 7.x will be an object and work with the above.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 22, 2019, 8:29am UTC](https://discuss.elastic.co/t/custom-haproxy-module-grok-pattern-failing/172613/3 "2019-04-22T08:29:50Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.