# Apache error log with referer

**URL:** <https://discuss.elastic.co/t/apache-error-log-with-referer/105770>\
**Category:** Logstash\
**Created:** [October 30, 2017, 2:25pm UTC](https://discuss.elastic.co/t/apache-error-log-with-referer/105770 "2017-10-30T14:25:25Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![jbelien](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jbelien/32/23562_2.png) [@jbelien](https://discuss.elastic.co/u/jbelien)\
**Post date:** [October 30, 2017, 2:25pm UTC](https://discuss.elastic.co/t/apache-error-log-with-referer/105770/1 "2017-10-30T14:25:25Z")

</div>

Hello everyone,

I have this line in my Apache error log :

```
[Fri Oct 20 16:39:12.472720 2017] [proxy_fcgi:error] [pid 5727:tid 124259626391296] [client 12.34.56.78:12345] AH01071: Got error 'PHP message: PHP Notice: Undefined index: nis5 in /var/www/fn.php on line 305\n', referer: https://mywebsite/

```

Every field is correctly extracted with `HTTPD24_ERRORLOG` as defined in [https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/httpd](https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/httpd) ;  
**But** I would like to also extract the `referer` from the end of the error message.

This `referer` is of course not always there so I tried this :

```
HTTPD24_ERRORLOG \[%{HTTPDERROR_DATE:timestamp}\] \[%{WORD:module}:%{LOGLEVEL:loglevel}\] \[pid %{POSINT:pid}(:tid %{NUMBER:tid})?\]( \(%{POSINT:proxy_errorcode}\)%{DATA:proxy_message}:)?( \[client %{IPORHOST:clientip}:%{POSINT:clientport}\])?( %{DATA:errorcode}:)? %{GREEDYDATA:message}(, referer: %{GREEDYDATA:referrer})?

```

But it doesn't work, the `referer` is not extracted and is still in `message` !  
How can I extract this `referer` ?

Thanks for your help !

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 30, 2017, 2:41pm UTC](https://discuss.elastic.co/t/apache-error-log-with-referer/105770/2 "2017-10-30T14:41:44Z")

</div>

This is a great example of why multiple DATA and/or GREEDYDATA is dangerous. In this case `%{GREEDYDATA:message}` gobbles up the rest of the message since the "referer: ..." stuff is optional. Try `%{DATA:message}` instead.

---

<div class="post-metadata">

**Author:** ![jbelien](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jbelien/32/23562_2.png) [@jbelien](https://discuss.elastic.co/u/jbelien)\
**Post date:** [October 30, 2017, 3:04pm UTC](https://discuss.elastic.co/t/apache-error-log-with-referer/105770/3 "2017-10-30T15:04:40Z")

</div>

Thanks a lot, I'll try that ! 🙂

---

<div class="post-metadata">

**Author:** ![jbelien](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jbelien/32/23562_2.png) [@jbelien](https://discuss.elastic.co/u/jbelien)\
**Post date:** [October 31, 2017, 8:10am UTC](https://discuss.elastic.co/t/apache-error-log-with-referer/105770/4 "2017-10-31T08:10:33Z")

</div>

Unfortunately, it does not seem to work !

`, referer: https://mywebsite/` is still included in **message** and not extracted as **referrer**.  
Is it possible to use something like `a+?` to match as few as possible ?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 31, 2017, 8:12am UTC](https://discuss.elastic.co/t/apache-error-log-with-referer/105770/5 "2017-10-31T08:12:49Z")

</div>

In this case I suggest using two grok expressions in the same filter (see example in the docs). The first one requires `, referer: %{GREEDYDATA:referrer}` at the end and the other one ends with `%{GREEDYDATA:message}`.

---

<div class="post-metadata">

**Author:** ![jbelien](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jbelien/32/23562_2.png) [@jbelien](https://discuss.elastic.co/u/jbelien)\
**Post date:** [October 31, 2017, 9:12am UTC](https://discuss.elastic.co/t/apache-error-log-with-referer/105770/6 "2017-10-31T09:12:09Z")

</div>

Awesome !

Something like this ?

```auto
grok {
  break_on_match => true
  match => { "message" => ["%{HTTPD24_ERRORLOG_REFERRER}", "%{HTTPD_ERRORLOG}"] } 
  patterns_dir => "/opt/logstash/patterns"
}

```

```auto
# Error logs
HTTPD20_ERRORLOG \[%{HTTPDERROR_DATE:timestamp}\] \[%{LOGLEVEL:loglevel}\] (?:\[client %{IPORHOST:clientip}\] ){0,1}%{GREEDYDATA:message}
HTTPD24_ERRORLOG \[%{HTTPDERROR_DATE:timestamp}\] \[%{WORD:module}:%{LOGLEVEL:loglevel}\] \[pid %{POSINT:pid}(:tid %{NUMBER:tid})?\]( \(%{POSINT:proxy_errorcode}\)%{DATA:proxy_message}:)?( \[client %{IPORHOST:clientip}:%{POSINT:clientport}\])?( %{DATA:errorcode}:)? %{GREEDYDATA:message}
HTTPD_ERRORLOG %{HTTPD20_ERRORLOG}|%{HTTPD24_ERRORLOG}

HTTPD24_ERRORLOG_REFERRER \[%{HTTPDERROR_DATE:timestamp}\] \[%{WORD:module}:%{LOGLEVEL:loglevel}\] \[pid %{POSINT:pid}(:tid %{NUMBER:tid})?\]( \(%{POSINT:proxy_errorcode}\)%{DATA:proxy_message}:)?( \[client %{IPORHOST:clientip}:%{POSINT:clientport}\])?( %{DATA:errorcode}:)? %{DATA:message}, referer: %{GREEDYDATA:referrer}

```

Thanks for all the help 🙂

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 31, 2017, 10:06am UTC](https://discuss.elastic.co/t/apache-error-log-with-referer/105770/7 "2017-10-31T10:06:27Z")

</div>

Yeah, that should work.

---

<div class="post-metadata">

**Author:** ![jbelien](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jbelien/32/23562_2.png) [@jbelien](https://discuss.elastic.co/u/jbelien)\
**Post date:** [October 31, 2017, 10:37am UTC](https://discuss.elastic.co/t/apache-error-log-with-referer/105770/8 "2017-10-31T10:37:17Z")

</div>

Unfortunately, that doesn't seem to work either ...

Filter:

```auto
grok {	
	break_on_match => true
	match => { "message" => ["%{OVHHTTPD_ERRORLOG_REFERRER}", "%{OVHHTTPD_ERRORLOG}"] } 
	patterns_dir => "/opt/logstash/patterns"
}

```

Grok:

```auto
OVHHTTPD20_ERRORLOG \[%{HTTPDERROR_DATE:timestamp}\] \[%{LOGLEVEL:loglevel}\] (?:\[client %{IPORHOST:clientip}\] ){0,1}%{GREEDYDATA:message}
OVHHTTPD24_ERRORLOG \[%{HTTPDERROR_DATE:timestamp}\] \[%{WORD:module}:%{LOGLEVEL:loglevel}\] \[pid %{POSINT:pid}(:tid %{NUMBER:tid})?\]( \(%{POSINT:proxy_errorcode}\)%{DATA:proxy_message}:)?( \[client %{IPORHOST:clientip}:%{POSINT:clientport}\])?( %{DATA:errorcode}:)? %{GREEDYDATA:message}
OVHHTTPD_ERRORLOG %{OVHHTTPD20_ERRORLOG}|%{OVHHTTPD24_ERRORLOG}

OVHHTTPD20_ERRORLOG_REFERRER \[%{HTTPDERROR_DATE:timestamp}\] \[%{LOGLEVEL:loglevel}\] (?:\[client %{IPORHOST:clientip}\] ){0,1}%{DATA:message}, referer: %{GREEDYDATA:referrer}
OVHHTTPD24_ERRORLOG_REFERRER \[%{HTTPDERROR_DATE:timestamp}\] \[%{WORD:module}:%{LOGLEVEL:loglevel}\] \[pid %{POSINT:pid}(:tid %{NUMBER:tid})?\]( \(%{POSINT:proxy_errorcode}\)%{DATA:proxy_message}:)?( \[client %{IPORHOST:clientip}:%{POSINT:clientport}\])?( %{DATA:errorcode}:)? %{DATA:message}, referer: %{GREEDYDATA:referrer}
OVHHTTPD_ERRORLOG_REFERRER %{OVHHTTPD20_ERRORLOG_REFERRER}|%{OVHHTTPD24_ERRORLOG_REFERRER}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 28, 2017, 10:37am UTC](https://discuss.elastic.co/t/apache-error-log-with-referer/105770/9 "2017-11-28T10:37:47Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.