# Logstash http server custom log format which includes values set in header

**URL:** <https://discuss.elastic.co/t/logstash-http-server-custom-log-format-which-includes-values-set-in-header/84647>\
**Category:** Logstash\
**Created:** [May 5, 2017, 3:59am UTC](https://discuss.elastic.co/t/logstash-http-server-custom-log-format-which-includes-values-set-in-header/84647 "2017-05-05T03:59:44Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![rajsolanki](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rajsolanki/32/86859_2.png) [@rajsolanki](https://discuss.elastic.co/u/rajsolanki)\
**Post date:** [May 5, 2017, 3:59am UTC](https://discuss.elastic.co/t/logstash-http-server-custom-log-format-which-includes-values-set-in-header/84647/1 "2017-05-05T03:59:44Z")

</div>

Hi

I have following LogFormat for my apache http server.

LogFormat "%h %l %u %t "%r" %\>s %b %D "%{Referer}i" "%{User-Agent}i" "%{JWT-elapsed-time}i" "%{userID}i" "%{userRole}o" "%{localServiceAddress}i" "%{X-Archived-Client-IP}i" "%{X-Forwarded-For}i" "%{operationID}i" " combined

\*\* We are using %{VARNAME}i The contents of VARNAME: header line(s) in the request sent to the server.\*\*

here is my one of sample log entry

11.11.111.11 - - [02/May/2017:12:07:52 -0500] "POST /myApp/myService HTTP/1.1" 200 53201 76578 "-" "-" "-" "myuserID" "-" "mylocalServiceAddress" "myX-Archived-Client-IP" "X-Forwarded-For" "myoperationID"

in grok debugger [http://grokdebug.herokuapp.com/](http://grokdebug.herokuapp.com/) i can make my pattern work till agent type.

%{IPORHOST:clientip} %{USER:ident} %{USER:auth} [%{HTTPDATE:timestamp}] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{NUMBER:timetoserve} %{QS:referrer} %{QS:agent}

how do i get last 5 fields which are %{VARNAME}i type ?

---

<div class="post-metadata">

**Author:** ![rajsolanki](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rajsolanki/32/86859_2.png) [@rajsolanki](https://discuss.elastic.co/u/rajsolanki)\
**Post date:** [May 6, 2017, 12:36am UTC](https://discuss.elastic.co/t/logstash-http-server-custom-log-format-which-includes-values-set-in-header/84647/2 "2017-05-06T00:36:19Z")

</div>

update

i m able to make following work.

%{IPORHOST:clientip} %{USER:ident} %{USER:auth} [%{HTTPDATE:timestamp}] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{NUMBER:timetoserve} %{QS:referrer} %{QS:agent} %{QS:AAA-elapsed-time} %{QS:userID} %{QS:userRole} %{QS:localServiceAddress}

but still 2 issues

1. despite of %{NUMBER:timetoserve} field is showing up as string
2. i m still not able to get last 3 fields ☹

please any one help ☹

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [May 8, 2017, 5:18am UTC](https://discuss.elastic.co/t/logstash-http-server-custom-log-format-which-includes-values-set-in-header/84647/3 "2017-05-08T05:18:49Z")

</div>

> despite of %{NUMBER:timetoserve} field is showing up as string

That's expected. You need `%{NUMBER:timetoserve:int}` to make the field an integer.

> i m still not able to get last 3 fields

You mean adding a couple of more `%{QS:whatever}` results in `_grokparsefailure`? Show us what configuration doesn't work and for what input it doesn't work.

---

<div class="post-metadata">

**Author:** ![rajsolanki](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rajsolanki/32/86859_2.png) [@rajsolanki](https://discuss.elastic.co/u/rajsolanki)\
**Post date:** [May 8, 2017, 12:38pm UTC](https://discuss.elastic.co/t/logstash-http-server-custom-log-format-which-includes-values-set-in-header/84647/4 "2017-05-08T12:38:53Z")

</div>

Thanks for reply. After doing research all weekend i was able to fix my problem. 🙂

1. had to create field in my template as int
2. %{IPORHOST:clientip} %{USER:ident} %{USER:auth} [%{HTTPDATE:timestamp}] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{NUMBER:responsetime} %{QS:referrer} %{QS:agent} %{QS:AAA-elapsed-time} %{QS:userID} %{QS:userRole} %{QS:localServiceAddress} %{GREEDYDATA:message} %{QS:operation}' }  
so above grok patterns works for me.

now i have another use case which has input like following

11.11.111.111 - - [07/May/2017:20:25:51 -0500] "GET /myage/page/my-panel-frag?tzo=1494206751935&tzn=America/Chicago HTTP/1.1" 200 3509 2720054 "[https://test-test.mydomain.com/myoperations/mypage/home](https://test-test.mydomain.com/myoperations/mypage/home)" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:53.0) Gecko/20100101 Firefox/53.0"

my grok pattern works till here

%{IPORHOST:clientip} %{USER:ident} %{USER:auth} [%{HTTPDATE:timestamp}] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{NUMBER:responsetime}

moment i add %{QS:referrer} it fails ☹

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 5, 2017, 12:50pm UTC](https://discuss.elastic.co/t/logstash-http-server-custom-log-format-which-includes-values-set-in-header/84647/5 "2017-06-05T12:50:54Z")

</div>

Closed on June 5, 2017, 12:50pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.