Need help in creating grok patterns for haproxy logs

nareshreddyp · June 1, 2018, 9:05am

Hello,

I am new to logstash and trying to filter the haproxy logs.

Example of my haproxy logs.

<134>0 2018-05-29T03:45:29+02:00 localhost epic-webservicefrontend 18645 - [type=haproxy] [18645] [1527558316.602] 3/3/1/0/0/0/0 5/0/0/4454/12916/{|gwlprod|Rjg2MzYyRDBDQTQ0NEEzREI3OUQzRDM0QjU5QzcyRDB8ZXBpY19sdHxlcGljX2x0fHx8MHw=|5b946a6b-2663-4c03-b31b-3938833514c3}/ --NI 128.87.242.20:39120 128.87.242.31:443 128.87.242.25:8011 https-in~ RequestCookies=- | "POST /3dspace/ericsson_services/Product?WSDL HTTP/1.1" 200 | backend_pool_sit2_3dspacebatch:SIT2_3dspace_BWS_Front_1

My Grok Expression:

grok {
match => ["message", "%{NOTSPACE:string} %{TIMESTAMP_ISO8601:timestamp8601} %{IPORHOST:syslog_server} %{SYSLOGPROG:serviceend} %{INT:HaproxyPID} - %{NOTSPACE:type} %{INT:Pid} %{INT:Ts}:%{INT:ms} %{INT:ac}/%{INT:fc}/%{INT:bc}/%{INT:bq}/%{INT:sc}/%{INT:sq}%{INT:rc} %{INT:Tq}/%{INT:Tw}/%{INT:Tc}/%{INT:Tr}/%{INT:Tt}/{|%{USER:http_user}|%{DATA:Headers}/%{INT:hs} %{DATA:tsc} %{IP:client_ip}:%{INT:client_port} %{IP:frontend_ip}:%{INT:frontend_port} %{IP:server_ip}:%{INT:server_port} %{DATA:transfer_type} [RequestCookies=- | %{DATA:cookie} %{INT:http_status_code} |]%{NOTSPACE:backendserver}:%{NOTSPACE:backendservice}"]
}

Result while starting logstash.

[2018-06-01T11:02:07,611][ERROR][logstash.pipeline ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<RegexpError: empty range in char class: /(?NOTSPACE:string\S+) (?<TIMESTAMP_ISO8601:timestamp8601>(?:(?>\d\d){1,2})-(?:(?:0?[1-9]|1[0-2]))-(?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))T :?(?:(?:[0-5][0-9]))(?::?(?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)))?(?:(?:Z|+-(?::?(?:(?:[0-5][0-9])))))?) (?IPORHOST:syslog_server(?:(?:(?:(?:((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:)))(%.+)?)|(?:(?<![0-9])(?:(?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])...)(?![0-9]))))|(?:\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))(.?|\b)))) (?SYSLOGPROG:serviceend(?PROG:program[\x21-\x5A\x5C\x5E-\x7E]+)(?:[(?POSINT:pid\b(?:[1-9][0-9])\b)])?) (?INT:HaproxyPID(?:[+-]?(?:[0-9]+))) - (?NOTSPACE:type\S+) (?INT:Pid(?:[+-]?(?:[0-9]+))) (?INT:Ts(?:[+-]?(?:[0-9]+))):(?INT:ms(?:[+-]?(?:[0-9]+))) (?INT:ac(?:[+-]?(?:[0-9]+)))/(?INT:fc(?:[+-]?(?:[0-9]+)))/(?INT:bc(?:[+-]?(?:[0-9]+)))/(?INT:bq(?:[+-]?(?:[0-9]+)))/(?INT:sc(?:[+-]?(?:[0-9]+)))/(?INT:sq(?:[+-]?(?:[0-9]+)))(?INT:rc(?:[+-]?(?:[0-9]+))) (?INT:Tq(?:[+-]?(?:[0-9]+)))/(?INT:Tw(?:[+-]?(?:[0-9]+)))/(?INT:Tc(?:[+-]?(?:[0-9]+)))/(?INT:Tr(?:[+-]?(?:[0-9]+)))/(?INT:Tt(?:[+-]?(?:[0-9]+)))/{|(?USER:http_user(?:[a-zA-Z0-9._-]+))|(?<DATA:Headers>.?)/(?INT:hs(?:[+-]?(?:[0-9]+))) (?<DATA:tsc>.?) (?IP:client_ip(?:(?:((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:)))(%.+)?)|(?:(?<![0-9])(?:(?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])...)(?![0-9])))):(?INT:client_port(?:[+-]?(?:[0-9]+))) (?IP:frontend_ip(?:(?:((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-

dyl · June 1, 2018, 10:03am

what field do you want to keep in your log ?

I mean, on this line, what fields are important for you ?

<134>0 2018-05-29T03:45:29+02:00 localhost epic-webservicefrontend 18645 - [type=haproxy] [18645] [1527558316.602] 3/3/1/0/0/0/0 5/0/0/4454/12916/{|gwlprod|Rjg2MzYyRDBDQTQ0NEEzREI3OUQzRDM0QjU5QzcyRDB8ZXBpY19sdHxlcGljX2x0fHx8MHw=|5b946a6b-2663-4c03-b31b-3938833514c3}/ --NI 128.87.242.20:39120 128.87.242.31:443 128.87.242.25:8011 https-in~ RequestCookies=- | "POST /3dspace/ericsson_services/Product?WSDL HTTP/1.1" 200 | backend_pool_sit2_3dspacebatch:SIT2_3dspace_BWS_Front_1

nareshreddyp · June 1, 2018, 10:35am

i need to filter all these fields and send it to elastic search / kibana

dyl · June 1, 2018, 11:54am

Ok but, filter how ?

What field to you want ?

For exemple in your message log line :

"number" => "<134>"
"Date" => "2018-05-29"
"Hour" => "03:45:29"
.
.
.

Tell me exactly what you expect

nareshreddyp · June 1, 2018, 12:19pm

string=<134>0
timestamp=2018-05-29T03:45:29+02:00
hostname=localhost
servicename=epic-webservicefrontend
processid=18645
string -
proxytype=[type=haproxy]
process=[18645]
time=[1527558316.602]
for this 3/3/1/0/0/0/0
ac=3
fc=3
bc=1
bq=0
sc=0
sq=0
rc=0
for this 5/0/0/4454/12916/{|gwlprod|rjg2mzyyrdbdqtq0neezrei3ouqzrdm0qju5qzcyrdb8zxbpy19sdhxlcgljx2x0fhx8mhw=|5b946a6b-2663-4c03-b31b-3938833514c3}/
Tq=5
Tw=0
Tc=0
Tr=4454
Tt=12916
username=gwlprod
Loginticket=Rjg2MzYyRDBDQTQ0NEEzREI3OUQzRDM0QjU5QzcyRDB8ZXBpY19sdHxlcGljX2x0fHx8MHw=
Header=5b946a6b-2663-4c03-b31b-3938833514c3
hs=

tsc= --NI
client_ip:client_port=128.87.242.20:39120
frontend_ip:frontend_port=128.87.242.31:443
server_ip:server_port=128.87.242.25:8011
transfer_type=https-in~

for this POST /3dspace/ericsson_services/Product?WSDL HTTP/1.1" 200
Request_type=POST
HTTP_URL= /3dspace/ericsson_services/Product?WSDL
HTTP_TYPE= HTTP/1.1
HTTP_STATUS_CODE=200
backendserver:backendservice=backend_pool_sit2_3dspacebatch:SIT2_3dspace_BWS_Front_1

dyl · June 1, 2018, 12:51pm

So ok go to : https://grokdebug.herokuapp.com/

And paste your log on the top window and begin you grok parse on the bottom window like this for you : <%{NUMBER:string}>0 %{TIMESTAMP_ISO8601:timestamp} %{WORD:hostname}

Check case "Named Captures Only" and "Singles"

And you are able to see in real time if your grok pattern if correct or note. Here a liste of syntax pattern :

github.com

elastic/logstash/blob/v1.4.0/patterns/grok-patterns

USERNAME [a-zA-Z0-9._-]+
USER %{USERNAME}
INT (?:[+-]?(?:[0-9]+))
BASE10NUM (?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))
NUMBER (?:%{BASE10NUM})
BASE16NUM (?<![0-9A-Fa-f])(?:[+-]?(?:0x)?(?:[0-9A-Fa-f]+))
BASE16FLOAT \b(?<![0-9A-Fa-f.])(?:[+-]?(?:0x)?(?:(?:[0-9A-Fa-f]+(?:\.[0-9A-Fa-f]*)?)|(?:\.[0-9A-Fa-f]+)))\b

POSINT \b(?:[1-9][0-9]*)\b
NONNEGINT \b(?:[0-9]+)\b
WORD \b\w+\b
NOTSPACE \S+
SPACE \s*
DATA .*?
GREEDYDATA .*
QUOTEDSTRING (?>(?<!\\)(?>"(?>\\.|[^\\"]+)+"|""|(?>'(?>\\.|[^\\']+)+')|''|(?>`(?>\\.|[^\\`]+)+`)|``))
UUID [A-Fa-f0-9]{8}-(?:[A-Fa-f0-9]{4}-){3}[A-Fa-f0-9]{12}

# Networking
MAC (?:%{CISCOMAC}|%{WINDOWSMAC}|%{COMMONMAC})

This file has been truncated. show original

nareshreddyp · June 1, 2018, 2:57pm

Hi,

Thank you very much for the link. I am not able to get the patterns for the following

{|gwlprod|rjg2mzyyrdbdqtq0neezrei3ouqzrdm0qju5qzcyrdb8zxbpy19sdhxlcgljx2x0fhx8mhw=|5b946a6b-2663-4c03-b31b-3938833514c3}
username=gwlprod
ticket =rjg2mzyyrdbdqtq0neezrei3ouqzrdm0qju5qzcyrdb8zxbpy19sdhxlcgljx2x0fhx8mhw=
header=5b946a6b-2663-4c03-b31b-3938833514c3

RequestCookies=- | "POST /3dspace/ericsson_services/Product?WSDL HTTP/1.1" 200 |
Method=Post
URL=/3dspace/ericsson_services/Product?WSDL
http_type=1.1
http response code=200

Can you please help guide me how to grok those ?

Regards,
Naresh

nareshreddyp · June 1, 2018, 4:08pm

Finally able to get it. Thank you very much for your support.

<%{NUMBER:number}>0 %{TIMESTAMP_ISO8601:timestamp} %{WORD:hostname} (%{WORD:service}-%{WORD:ServiceName}) %{INT:Process} %{PROG:string1} [type=%{PROG:haproxy}] [%{INT:processid}] [%{GREEDYDATA:unixtimestamp}] %{INT:ac}/%{INT:fc}/%{INT:bc}/%{INT:bq}/%{INT:sc}/%{INT:sq}/%{INT:rc} %{INT:Tq}/%{INT:Tw}/%{INT:Tc}/%{INT:Tr}/%{INT:Tt}/({|%{USER:username}|%{GREEDYDATA:LoginTicket}=|%{GREEDYDATA:header}}/) %{PROG:tws} %{IP:ClientIP}:%{INT:ClientPort} %{IP:FrontendIP}:%{INT:FrontendPort} %{IP:BackendIP}:%{INT:BackendPort} %{PROG:Transport_Type}~ RequestCookies=- | "%{WORD:Method} %{URIPATHPARAM:request} HTTP/%{NUMBER:http_version}" %{INT:HTTP_Response_Code} | %{WORD:backend}:%{WORD:backendserver}

system · June 29, 2018, 4:08pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Help with grok pattern for haproxy Logstash	4	1883	February 19, 2020
How to parse Haproxy log with grok Logstash	6	6129	January 3, 2019
Logstash two different log types Logstash	8	870	June 26, 2018
How to grok haproxy Log Logstash	4	476	January 3, 2023
Grok pattern not working Logstash	6	773	July 24, 2020

Need help in creating grok patterns for haproxy logs

Related topics