# Need help in creating grok patterns for haproxy logs

**URL:** <https://discuss.elastic.co/t/need-help-in-creating-grok-patterns-for-haproxy-logs/134180>\
**Category:** Logstash\
**Created:** [June 1, 2018, 9:05am UTC](https://discuss.elastic.co/t/need-help-in-creating-grok-patterns-for-haproxy-logs/134180 "2018-06-01T09:05:20Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![nareshreddyp](https://avatars.discourse-cdn.com/v4/letter/n/e56c9b/32.png) [@nareshreddyp](https://discuss.elastic.co/u/nareshreddyp)\
**Post date:** [June 1, 2018, 9:05am UTC](https://discuss.elastic.co/t/need-help-in-creating-grok-patterns-for-haproxy-logs/134180/1 "2018-06-01T09:05:20Z")

</div>

Hello,

I am new to logstash and trying to filter the haproxy logs.

Example of my haproxy logs.

\<134\>0 2018-05-29T03:45:29+02:00 localhost epic-webservicefrontend 18645 - [type=haproxy] [18645] [1527558316.602] 3/3/1/0/0/0/0 5/0/0/4454/12916/{|gwlprod|Rjg2MzYyRDBDQTQ0NEEzREI3OUQzRDM0QjU5QzcyRDB8ZXBpY19sdHxlcGljX2x0fHx8MHw=|5b946a6b-2663-4c03-b31b-3938833514c3}/ --NI 128.87.242.20:39120 128.87.242.31:443 128.87.242.25:8011 https-in~ RequestCookies=- | "POST /3dspace/ericsson\_services/Product?WSDL HTTP/1.1" 200 | backend\_pool\_sit2\_3dspacebatch:SIT2\_3dspace\_BWS\_Front\_1

My Grok Expression:

grok {  
match =\> ["message", "%{NOTSPACE:string} %{TIMESTAMP\_ISO8601:timestamp8601} %{IPORHOST:syslog\_server} %{SYSLOGPROG:serviceend} %{INT:HaproxyPID} - %{NOTSPACE:type} %{INT:Pid} %{INT:Ts}:%{INT:ms} %{INT:ac}/%{INT:fc}/%{INT:bc}/%{INT:bq}/%{INT:sc}/%{INT:sq}%{INT:rc} %{INT:Tq}/%{INT:Tw}/%{INT:Tc}/%{INT:Tr}/%{INT:Tt}/{|%{USER:http\_user}|%{DATA:Headers}/%{INT:hs} %{DATA:tsc} %{IP:client\_ip}:%{INT:client\_port} %{IP:frontend\_ip}:%{INT:frontend\_port} %{IP:server\_ip}:%{INT:server\_port} %{DATA:transfer\_type} [RequestCookies=- | %{DATA:cookie} %{INT:http\_status\_code} |]%{NOTSPACE:backendserver}:%{NOTSPACE:backendservice}"]  
}

Result while starting logstash.

[2018-06-01T11:02:07,611][ERROR][logstash.pipeline] Pipeline aborted due to error {:pipeline\_id=\>"main", :exception=\>#\<RegexpError: empty range in char class: /(?NOTSPACE:string\S+) (?\<TIMESTAMP\_ISO8601:timestamp8601\>(?:(?\>\d\d){1,2})-(?:(?:0?[1-9]|1[0-2]))-(?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))T :?(?:(?:[0-5][0-9]))(?::?(?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)))?(?:(?:Z|+-(?::?(?:(?:[0-5][0-9])))))?) (?IPORHOST:syslog\_server(?:(?:(?:(?:((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:)))(%.+)?)|(?:(?\<![0-9])(?:(?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])...)(?![0-9]))))|(?:\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))_(.?|\b)))) (?SYSLOGPROG:serviceend(?PROG:program[\x21-\x5A\x5C\x5E-\x7E]+)(?:[(?POSINT:pid\b(?:[1-9][0-9]_)\b)])?) (?INT:HaproxyPID(?:[+-]?(?:[0-9]+))) - (?NOTSPACE:type\S+) (?INT:Pid(?:[+-]?(?:[0-9]+))) (?INT:Ts(?:[+-]?(?:[0-9]+))):(?INT:ms(?:[+-]?(?:[0-9]+))) (?INT:ac(?:[+-]?(?:[0-9]+)))/(?INT:fc(?:[+-]?(?:[0-9]+)))/(?INT:bc(?:[+-]?(?:[0-9]+)))/(?INT:bq(?:[+-]?(?:[0-9]+)))/(?INT:sc(?:[+-]?(?:[0-9]+)))/(?INT:sq(?:[+-]?(?:[0-9]+)))(?INT:rc(?:[+-]?(?:[0-9]+))) (?INT:Tq(?:[+-]?(?:[0-9]+)))/(?INT:Tw(?:[+-]?(?:[0-9]+)))/(?INT:Tc(?:[+-]?(?:[0-9]+)))/(?INT:Tr(?:[+-]?(?:[0-9]+)))/(?INT:Tt(?:[+-]?(?:[0-9]+)))/{|(?USER:http\_user(?:[a-zA-Z0-9.\_-]+))|(?\<DATA:Headers\>._?)/(?INT:hs(?:[+-]?(?:[0-9]+))) (?\<DATA:tsc\>._?) (?IP:client\_ip(?:(?:((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:)))(%.+)?)|(?:(?\<![0-9])(?:(?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])...)(?![0-9])))):(?INT:client\_port(?:[+-]?(?:[0-9]+))) (?IP:frontend\_ip(?:(?:((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-

---

<div class="post-metadata">

**Author:** ![dyl](https://avatars.discourse-cdn.com/v4/letter/d/5fc32e/32.png) [@dyl](https://discuss.elastic.co/u/dyl)\
**Post date:** [June 1, 2018, 10:03am UTC](https://discuss.elastic.co/t/need-help-in-creating-grok-patterns-for-haproxy-logs/134180/2 "2018-06-01T10:03:06Z")

</div>

what field do you want to keep in your log ?

I mean, on this line, what fields are important for you ?

> \<134\>0 2018-05-29T03:45:29+02:00 localhost epic-webservicefrontend 18645 - [type=haproxy] [18645] [1527558316.602] 3/3/1/0/0/0/0 5/0/0/4454/12916/{|gwlprod|Rjg2MzYyRDBDQTQ0NEEzREI3OUQzRDM0QjU5QzcyRDB8ZXBpY19sdHxlcGljX2x0fHx8MHw=|5b946a6b-2663-4c03-b31b-3938833514c3}/ --NI 128.87.242.20:39120 128.87.242.31:443 128.87.242.25:8011 https-in~ RequestCookies=- | "POST /3dspace/ericsson\_services/Product?WSDL HTTP/1.1" 200 | backend\_pool\_sit2\_3dspacebatch:SIT2\_3dspace\_BWS\_Front\_1

---

<div class="post-metadata">

**Author:** ![nareshreddyp](https://avatars.discourse-cdn.com/v4/letter/n/e56c9b/32.png) [@nareshreddyp](https://discuss.elastic.co/u/nareshreddyp)\
**Post date:** [June 1, 2018, 10:35am UTC](https://discuss.elastic.co/t/need-help-in-creating-grok-patterns-for-haproxy-logs/134180/3 "2018-06-01T10:35:42Z")

</div>

i need to filter all these fields and send it to elastic search / kibana

---

<div class="post-metadata">

**Author:** ![dyl](https://avatars.discourse-cdn.com/v4/letter/d/5fc32e/32.png) [@dyl](https://discuss.elastic.co/u/dyl)\
**Post date:** [June 1, 2018, 11:54am UTC](https://discuss.elastic.co/t/need-help-in-creating-grok-patterns-for-haproxy-logs/134180/4 "2018-06-01T11:54:51Z")

</div>

Ok but, filter how ?

What field to you want ?

For exemple in your message log line :

"number" =\> "\<134\>"  
"Date" =\> "2018-05-29"  
"Hour" =\> "03:45:29"  
.  
.  
.

Tell me exactly what you expect

---

<div class="post-metadata">

**Author:** ![nareshreddyp](https://avatars.discourse-cdn.com/v4/letter/n/e56c9b/32.png) [@nareshreddyp](https://discuss.elastic.co/u/nareshreddyp)\
**Post date:** [June 1, 2018, 12:19pm UTC](https://discuss.elastic.co/t/need-help-in-creating-grok-patterns-for-haproxy-logs/134180/5 "2018-06-01T12:19:45Z")

</div>

string=\<134\>0  
timestamp=2018-05-29T03:45:29+02:00  
hostname=localhost  
servicename=epic-webservicefrontend  
processid=18645  
string -  
proxytype=[type=haproxy]  
process=[18645]  
time=[1527558316.602]  
for this 3/3/1/0/0/0/0  
ac=3  
fc=3  
bc=1  
bq=0  
sc=0  
sq=0  
rc=0  
for this 5/0/0/4454/12916/{|gwlprod|rjg2mzyyrdbdqtq0neezrei3ouqzrdm0qju5qzcyrdb8zxbpy19sdhxlcgljx2x0fhx8mhw=|5b946a6b-2663-4c03-b31b-3938833514c3}/  
Tq=5  
Tw=0  
Tc=0  
Tr=4454  
Tt=12916  
username=gwlprod  
Loginticket=Rjg2MzYyRDBDQTQ0NEEzREI3OUQzRDM0QjU5QzcyRDB8ZXBpY19sdHxlcGljX2x0fHx8MHw=  
Header=5b946a6b-2663-4c03-b31b-3938833514c3  
hs=

tsc= --NI  
client\_ip:client\_port=128.87.242.20:39120  
frontend\_ip:frontend\_port=128.87.242.31:443  
server\_ip:server\_port=128.87.242.25:8011  
transfer\_type=https-in~

for this POST /3dspace/ericsson\_services/Product?WSDL HTTP/1.1" 200  
Request\_type=POST  
HTTP\_URL= /3dspace/ericsson\_services/Product?WSDL  
HTTP\_TYPE= HTTP/1.1  
HTTP\_STATUS\_CODE=200  
backendserver:backendservice=backend\_pool\_sit2\_3dspacebatch:SIT2\_3dspace\_BWS\_Front\_1

---

<div class="post-metadata">

**Author:** ![dyl](https://avatars.discourse-cdn.com/v4/letter/d/5fc32e/32.png) [@dyl](https://discuss.elastic.co/u/dyl)\
**Post date:** [June 1, 2018, 12:51pm UTC](https://discuss.elastic.co/t/need-help-in-creating-grok-patterns-for-haproxy-logs/134180/6 "2018-06-01T12:51:08Z")

</div>

So ok go to : [https://grokdebug.herokuapp.com/](https://grokdebug.herokuapp.com/)

And paste your log on the top window and begin you grok parse on the bottom window like this for you : \<%{NUMBER:string}\>0 %{TIMESTAMP\_ISO8601:timestamp} %{WORD:hostname}

Check case "Named Captures Only" and "Singles"

And you are able to see in real time if your grok pattern if correct or note. Here a liste of syntax pattern :

> <https://github.com/elastic/logstash/blob/v1.4.0/patterns/grok-patterns>

---

<div class="post-metadata">

**Author:** ![nareshreddyp](https://avatars.discourse-cdn.com/v4/letter/n/e56c9b/32.png) [@nareshreddyp](https://discuss.elastic.co/u/nareshreddyp)\
**Post date:** [June 1, 2018, 2:57pm UTC](https://discuss.elastic.co/t/need-help-in-creating-grok-patterns-for-haproxy-logs/134180/7 "2018-06-01T14:57:52Z")

</div>

Hi,

Thank you very much for the link. I am not able to get the patterns for the following

{|gwlprod|rjg2mzyyrdbdqtq0neezrei3ouqzrdm0qju5qzcyrdb8zxbpy19sdhxlcgljx2x0fhx8mhw=|5b946a6b-2663-4c03-b31b-3938833514c3}  
username=gwlprod  
ticket =rjg2mzyyrdbdqtq0neezrei3ouqzrdm0qju5qzcyrdb8zxbpy19sdhxlcgljx2x0fhx8mhw=  
header=5b946a6b-2663-4c03-b31b-3938833514c3

RequestCookies=- | "POST /3dspace/ericsson\_services/Product?WSDL HTTP/1.1" 200 |  
Method=Post  
URL=/3dspace/ericsson\_services/Product?WSDL  
http\_type=1.1  
http response code=200

Can you please help guide me how to grok those ?

Regards,  
Naresh

---

<div class="post-metadata">

**Author:** ![nareshreddyp](https://avatars.discourse-cdn.com/v4/letter/n/e56c9b/32.png) [@nareshreddyp](https://discuss.elastic.co/u/nareshreddyp)\
**Post date:** [June 1, 2018, 4:08pm UTC](https://discuss.elastic.co/t/need-help-in-creating-grok-patterns-for-haproxy-logs/134180/8 "2018-06-01T16:08:40Z")

</div>

Finally able to get it. Thank you very much for your support.

\<%{NUMBER:number}\>0 %{TIMESTAMP\_ISO8601:timestamp} %{WORD:hostname} (%{WORD:service}-%{WORD:ServiceName}) %{INT:Process} %{PROG:string1} [type=%{PROG:haproxy}] [%{INT:processid}] [%{GREEDYDATA:unixtimestamp}] %{INT:ac}/%{INT:fc}/%{INT:bc}/%{INT:bq}/%{INT:sc}/%{INT:sq}/%{INT:rc} %{INT:Tq}/%{INT:Tw}/%{INT:Tc}/%{INT:Tr}/%{INT:Tt}/({|%{USER:username}|%{GREEDYDATA:LoginTicket}=|%{GREEDYDATA:header}}/) %{PROG:tws} %{IP:ClientIP}:%{INT:ClientPort} %{IP:FrontendIP}:%{INT:FrontendPort} %{IP:BackendIP}:%{INT:BackendPort} %{PROG:Transport\_Type}~ RequestCookies=- | "%{WORD:Method} %{URIPATHPARAM:request} HTTP/%{NUMBER:http\_version}" %{INT:HTTP\_Response\_Code} | %{WORD:backend}:%{WORD:backendserver}

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 29, 2018, 4:08pm UTC](https://discuss.elastic.co/t/need-help-in-creating-grok-patterns-for-haproxy-logs/134180/9 "2018-06-29T16:08:42Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.