Logstash http server custom log format which includes values set in header

[rajsolanki (rajesh)]

Hi

I have following LogFormat for my apache http server.

LogFormat "%h %l %u %t "%r" %>s %b %D "%{Referer}i" "%{User-Agent}i" "%{JWT-elapsed-time}i" "%{userID}i" "%{userRole}o" "%{localServiceAddress}i" "%{X-Archived-Client-IP}i" "%{X-Forwarded-For}i" "%{operationID}i" " combined

** We are using %{VARNAME}i The contents of VARNAME: header line(s) in the request sent to the server.**

here is my one of sample log entry

11.11.111.11 - - [02/May/2017:12:07:52 -0500] "POST /myApp/myService HTTP/1.1" 200 53201 76578 "-" "-" "-" "myuserID" "-" "mylocalServiceAddress" "myX-Archived-Client-IP" "X-Forwarded-For" "myoperationID"

in grok debugger http://grokdebug.herokuapp.com/ i can make my pattern work till agent type.

%{IPORHOST:clientip} %{USER:ident} %{USER:auth} [%{HTTPDATE:timestamp}] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{NUMBER:timetoserve} %{QS:referrer} %{QS:agent}

how do i get last 5 fields which are %{VARNAME}i type ?

[rajsolanki (rajesh)]

update

i m able to make following work.

%{IPORHOST:clientip} %{USER:ident} %{USER:auth} [%{HTTPDATE:timestamp}] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{NUMBER:timetoserve} %{QS:referrer} %{QS:agent} %{QS:AAA-elapsed-time} %{QS:userID} %{QS:userRole} %{QS:localServiceAddress}

but still 2 issues

1. despite of %{NUMBER:timetoserve} field is showing up as string
2. i m still not able to get last 3 fields

please any one help

[magnusbaeck (Magnus Bäck)]

despite of %{NUMBER:timetoserve} field is showing up as string

That's expected. You need %{NUMBER:timetoserve:int} to make the field an integer.

i m still not able to get last 3 fields

You mean adding a couple of more %{QS:whatever} results in _grokparsefailure? Show us what configuration doesn't work and for what input it doesn't work.

[rajsolanki (rajesh)]

Thanks for reply. After doing research all weekend i was able to fix my problem.

1. had to create field in my template as int
2. %{IPORHOST:clientip} %{USER:ident} %{USER:auth} [%{HTTPDATE:timestamp}] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{NUMBER:responsetime} %{QS:referrer} %{QS:agent} %{QS:AAA-elapsed-time} %{QS:userID} %{QS:userRole} %{QS:localServiceAddress} %{GREEDYDATA:message} %{QS:operation}' }
   so above grok patterns works for me.

now i have another use case which has input like following

11.11.111.111 - - [07/May/2017:20:25:51 -0500] "GET /myage/page/my-panel-frag?tzo=1494206751935&tzn=America/Chicago HTTP/1.1" 200 3509 2720054 "https://test-test.mydomain.com/myoperations/mypage/home" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:53.0) Gecko/20100101 Firefox/53.0"

my grok pattern works till here

%{IPORHOST:clientip} %{USER:ident} %{USER:auth} [%{HTTPDATE:timestamp}] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{NUMBER:responsetime}

moment i add %{QS:referrer} it fails