# Can't parse haproxy logs without IP address in Grok

**URL:** <https://discuss.elastic.co/t/cant-parse-haproxy-logs-without-ip-address-in-grok/273654>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [May 21, 2021, 10:57am UTC](https://discuss.elastic.co/t/cant-parse-haproxy-logs-without-ip-address-in-grok/273654 "2021-05-21T10:57:23Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![maar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maar/32/80269_2.png) [@maar](https://discuss.elastic.co/u/maar)\
**Post date:** [May 21, 2021, 10:57am UTC](https://discuss.elastic.co/t/cant-parse-haproxy-logs-without-ip-address-in-grok/273654/1 "2021-05-21T10:57:23Z")

</div>

Grok is parsing successfully when Haproxy gives a log - from `var/log/haproxy.log` - similar to:

```auto
May 21 08:25:56 ha haproxy[5089]: 12.3.45.67:89012 [21/May/2021:08:25:56.055] www-https~ wss/wssnode website.domain.com 1/1/1/1/111 111 111 - - ---- 11111/11111/11111/111/0 0/0 "GET /ws/site/V3L235F/d88r3567pssllp/ HTTP/1.1"

```

But when instead of `ip_address:port` there's a `-:port` , for example:

```auto
May 21 08:25:56 ha haproxy[5089]: -:89012 [21/May/2021:08:25:56.055] www-https~ wss/wssnode website.domain.com 1/1/1/1/111 111 111 - - ---- 11111/11111/11111/111/0 0/0 "GET /ws/site/V3L235F/d88r3567pssllp/ HTTP/1.1"

```

I have an error:

> Provided Grok expressions do not match field value: May 21 08:25:56 ha haproxy[5089]: -:89012 [21/May/2021:08:25:56.055] www-https~ wss/wssnode [website.domain.com](http://website.domain.com) 1/1/1/1/111 111 111 - - ---- 11111/11111/11111/111/0 0/0 "GET /ws/site/V3L235F/d88r3567pssllp/ HTTP/1.1

Here's my [/usr/share/filebeat/module/haproxy/log/pipline.json](https://gist.github.com/maarsaks/0d9b094da9e5747050be03184a031b16)

I was trying to resolve this by adding a new pattern to `grok` with `message` field and by editing `grok` pattern with `source.address` field without success.

---

<div class="post-metadata">

**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\
**Post date:** [May 21, 2021, 7:57pm UTC](https://discuss.elastic.co/t/cant-parse-haproxy-logs-without-ip-address-in-grok/273654/2 "2021-05-21T19:57:12Z")

</div>

Is this a repeated issue you're having? Any special config that causes the - instead of an IP address? This should be an easy update to the ingest pipeline. Can you make an issue on GitHub for tracking?

---

<div class="post-metadata">

**Author:** ![maar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maar/32/80269_2.png) [@maar](https://discuss.elastic.co/u/maar)\
**Post date:** [May 24, 2021, 7:47am UTC](https://discuss.elastic.co/t/cant-parse-haproxy-logs-without-ip-address-in-grok/273654/3 "2021-05-24T07:47:49Z")

</div>

> [@legoguy1000](#):
>
> Is this a repeated issue you're having?

Yes

> [@legoguy1000](#):
>
> Any special config that causes the - instead of an IP address?

No, default config. Haproxy doesn't intercept every log with IP addresses.

> [@legoguy1000](#):
>
> This should be an easy update to the ingest pipeline. Can you make an issue on GitHub for tracking?

Actually I already have the correct Grok expression that works in debugger, but still it doesn't work in Kibana. Maybe there's something else that causes the error.  
I'll create an issue on GitHub.

Thanks for your help.

---

<div class="post-metadata">

**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\
**Post date:** [May 24, 2021, 10:50am UTC](https://discuss.elastic.co/t/cant-parse-haproxy-logs-without-ip-address-in-grok/273654/4 "2021-05-24T10:50:12Z")

</div>

Also what version of filebeat and elasticsearch are you using?

---

<div class="post-metadata">

**Author:** ![maar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maar/32/80269_2.png) [@maar](https://discuss.elastic.co/u/maar)\
**Post date:** [May 24, 2021, 1:24pm UTC](https://discuss.elastic.co/t/cant-parse-haproxy-logs-without-ip-address-in-grok/273654/5 "2021-05-24T13:24:08Z")

</div>

Filebeat 7.6.1  
Elastic&Kibana 7.7.1

> <https://github.com/elastic/beats/issues/25827>
>
> I created Grok pattern which works in Kibana Debug Grok devtool:
> Sample data:
> …\`\`\`
> \[May 22 02:22:22 server1 haproxy\[5089\]: -:22222 \[22/May/2021:02:22:22.222\] www-https~ myapp/node2 site.domain.com 0/0/0/18/18 200 200 - - ---- 222/222/2/0/0 0/0 \\"OPTIONS /api/v2/app/ HTTP/1.1\\"\]
> \`\`\`
> Grok pattern:
> \`\`\`
> %{HAPROXY\_LOG\_DATE:haproxy.logdate} %{NOTSPACE:haproxy.host} %{NOTSPACE:process.name\[pid\]}\[%{NUMBER:process.pid:long}\] (%{IP:source.address}|-):%{POSINT:source} %{HAPROXY\_DATE:haproxy.request\_date} %{NOTSPACE:haproxy.frontend\_name} %{NOTSPACE:haproxy.backend\_name}/%{NOTSPACE:haproxy.server\_name} %{NOTSPACE:haproxy.http.captured.request.headers} %{NUMBER:haproxy.http.request.time\_wait\_ms:long}/%{NUMBER:haproxy.total\_waiting\_time\_ms:long}/%{NUMBER:haproxy.connection\_wait\_time\_ms:long}/%{NUMBER:haproxy.http.request.time\_wait\_without\_data\_ms:long}/%{NUMBER:temp.duration:long} %{NUMBER:http.response.status\_code:long} %{NUMBER:haproxy.bytes\_read:long} %{NOTSPACE:haproxy.http.request.captured\_cookie} %{NOTSPACE:haproxy.http.response.captured\_cookie} %{NOTSPACE:haproxy.termination\_state} %{NUMBER:haproxy.connections.active:long}/%{NUMBER:haproxy.connections.frontend:long}/%{NUMBER:haproxy.connections.backend:long}/%{NUMBER:haproxy.connections.server:long}/%{NUMBER:haproxy.connections.retries:long} %{NUMBER:haproxy.server\_queue:long}/%{NUMBER:haproxy.backend\_queue:long} \\\\\\"%{NOTSPACE:haproxy.http.request.method} %{NOTSPACE:haproxy.http.request.captured\_headers} %{NOTSPACE:haproxy.http.response.captured\_headers}\\\\\\"
> \`\`\`
> 
> Custom patterns:
> \`\`\`
> HAPROXY\_LOG\_DATE %{MONTH} %{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND}
> HAPROXY\_DATE \\\[%{MONTHDAY}\[/-\]%{MONTH}\[/-\]%{YEAR}:%{HOUR}:%{MINUTE}:%{SECOND}\\\]
> \`\`\`
> Structured data:
> \`\`\`
> {
> "process": {
> "name\[pid\]": "haproxy\[5089\]"
> },
> "temp": {
> "duration": 18
> },
> "haproxy": {
> "server\_name": "node2",
> "total\_waiting\_time\_ms": 0,
> "termination\_state": "----",
> "connection\_wait\_time\_ms": 0,
> "bytes\_read": 200,
> "backend\_queue": 0,
> "backend\_name": "myapp",
> "logdate": "May 22 02:22:22",
> "host": "server1",
> "request\_date": "\[22/May/2021:02:22:22.222\]",
> "http": {
> "request": {
> "captured\_cookie": "-",
> "time\_wait\_without\_data\_ms": 18,
> "captured\_headers": "/api/v2/app/",
> "method": "OPTIONS",
> "time\_wait\_ms": 0
> },
> "response": {
> "captured\_cookie": "-",
> "captured\_headers": "HTTP/1.1"
> },
> "captured": {
> "request": {
> "headers": "site.domain.com"
> }
> }
> },
> "frontend\_name": "www-https~",
> "server\_queue": 0,
> "connections": {
> "server": 0,
> "retries": 0,
> "active": 222,
> "backend": 2,
> "frontend": 222
> }
> },
> "http": {
> "response": {
> "status\_code": 200
> }
> },
> "source": "22222"
> }
> \`\`\`
> 
> First I thought that something else causes the problem: \[discuss.elastic.co\](https://discuss.elastic.co/t/cant-parse-haproxy-logs-without-ip-address-in-grok/273654)
> 
> But the pattern is working, and I don't know why. but it doesn't work with Filebeat and Elastic.
> I have an error in Elastic:
> \`\`\`
> Provided Grok expressions do not match field value:
> \[May 22 02:22:22 server1 haproxy\[5089\]: -:22222 \[22/May/2021:02:22:22.222\] www-https~ myapp/node2 site.domain.com 0/0/0/18/18 200 200 - - ---- 222/222/2/0/0 0/0 \\"OPTIONS /api/v2/app/ HTTP/1.1\\"\]
> 
> \`\`\`
> That's my current config file: \[gist.github.com\](https://gist.github.com/maarsaks/c7eb65a4d5e64681d6dd33c5898438b5)
> 
> 
> 
> \- Version: Filebeat 7.6.1; Elastic&Kibana 7.7.1
> \- Operating System: Debian Buster
> \- Discuss Forum URL: \[discuss.elastic.co\](https://discuss.elastic.co/t/cant-parse-haproxy-logs-without-ip-address-in-grok/273654)

---

<div class="post-metadata">

**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\
**Post date:** [May 25, 2021, 2:37am UTC](https://discuss.elastic.co/t/cant-parse-haproxy-logs-without-ip-address-in-grok/273654/6 "2021-05-25T02:37:41Z")

</div>

I created a PR, [[Filebeat] Update HA Proxy log grok patterns by legoguy1000 · Pull Request #25835 · elastic/beats · GitHub](https://github.com/elastic/beats/pull/25835), please take a look.

---

<div class="post-metadata">

**Author:** ![maar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maar/32/80269_2.png) [@maar](https://discuss.elastic.co/u/maar)\
**Post date:** [May 25, 2021, 9:48am UTC](https://discuss.elastic.co/t/cant-parse-haproxy-logs-without-ip-address-in-grok/273654/7 "2021-05-25T09:48:05Z")

</div>

Thanks. I've replied under the issue discussion.

---

<div class="post-metadata">

**Author:** ![maar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maar/32/80269_2.png) [@maar](https://discuss.elastic.co/u/maar)\
**Post date:** [June 9, 2021, 2:35pm UTC](https://discuss.elastic.co/t/cant-parse-haproxy-logs-without-ip-address-in-grok/273654/8 "2021-06-09T14:35:40Z")

</div>

I've created the repo how to reproduce the error in ~5 minutes - make sure that you've made the all steps from `README.md` .  
[github.com/maarsaks/elk-docker-compose](https://github.com/maarsaks/elk-docker-compose)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 7, 2021, 4:36pm UTC](https://discuss.elastic.co/t/cant-parse-haproxy-logs-without-ip-address-in-grok/273654/9 "2021-07-07T16:36:14Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.