Apache error log with referer

Elastic Stack Logstash
1

Hello everyone,

I have this line in my Apache error log :

[Fri Oct 20 16:39:12.472720 2017] [proxy_fcgi:error] [pid 5727:tid 124259626391296] [client 12.34.56.78:12345] AH01071: Got error 'PHP message: PHP Notice: Undefined index: nis5 in /var/www/fn.php on line 305\n', referer: https://mywebsite/

Every field is correctly extracted with HTTPD24_ERRORLOG as defined in https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/httpd ;
But I would like to also extract the referer from the end of the error message.

This referer is of course not always there so I tried this :

HTTPD24_ERRORLOG \[%{HTTPDERROR_DATE:timestamp}\] \[%{WORD:module}:%{LOGLEVEL:loglevel}\] \[pid %{POSINT:pid}(:tid %{NUMBER:tid})?\]( \(%{POSINT:proxy_errorcode}\)%{DATA:proxy_message}:)?( \[client %{IPORHOST:clientip}:%{POSINT:clientport}\])?( %{DATA:errorcode}:)? %{GREEDYDATA:message}(, referer: %{GREEDYDATA:referrer})?

But it doesn't work, the referer is not extracted and is still in message !
How can I extract this referer ?

Thanks for your help !

2

This is a great example of why multiple DATA and/or GREEDYDATA is dangerous. In this case %{GREEDYDATA:message} gobbles up the rest of the message since the "referer: ..." stuff is optional. Try %{DATA:message} instead.

1 Like
3

Thanks a lot, I'll try that ! :slight_smile:

4

Unfortunately, it does not seem to work !

, referer: https://mywebsite/ is still included in message and not extracted as referrer.
Is it possible to use something like a+? to match as few as possible ?

5

In this case I suggest using two grok expressions in the same filter (see example in the docs). The first one requires , referer: %{GREEDYDATA:referrer} at the end and the other one ends with %{GREEDYDATA:message}.

1 Like
6

Awesome !

Something like this ?

grok {
  break_on_match => true
  match => { "message" => [ "%{HTTPD24_ERRORLOG_REFERRER}", "%{HTTPD_ERRORLOG}" ] } 
  patterns_dir => "/opt/logstash/patterns"
}


# Error logs
HTTPD20_ERRORLOG \[%{HTTPDERROR_DATE:timestamp}\] \[%{LOGLEVEL:loglevel}\] (?:\[client %{IPORHOST:clientip}\] ){0,1}%{GREEDYDATA:message}
HTTPD24_ERRORLOG \[%{HTTPDERROR_DATE:timestamp}\] \[%{WORD:module}:%{LOGLEVEL:loglevel}\] \[pid %{POSINT:pid}(:tid %{NUMBER:tid})?\]( \(%{POSINT:proxy_errorcode}\)%{DATA:proxy_message}:)?( \[client %{IPORHOST:clientip}:%{POSINT:clientport}\])?( %{DATA:errorcode}:)? %{GREEDYDATA:message}
HTTPD_ERRORLOG %{HTTPD20_ERRORLOG}|%{HTTPD24_ERRORLOG}

HTTPD24_ERRORLOG_REFERRER \[%{HTTPDERROR_DATE:timestamp}\] \[%{WORD:module}:%{LOGLEVEL:loglevel}\] \[pid %{POSINT:pid}(:tid %{NUMBER:tid})?\]( \(%{POSINT:proxy_errorcode}\)%{DATA:proxy_message}:)?( \[client %{IPORHOST:clientip}:%{POSINT:clientport}\])?( %{DATA:errorcode}:)? %{DATA:message}, referer: %{GREEDYDATA:referrer}

Thanks for all the help :slight_smile:

7

Yeah, that should work.

1 Like
8

Unfortunately, that doesn't seem to work either ...

Filter:

grok {	
	break_on_match => true
	match => { "message" => [ "%{OVHHTTPD_ERRORLOG_REFERRER}", "%{OVHHTTPD_ERRORLOG}" ] } 
	patterns_dir => "/opt/logstash/patterns"
}

Grok:

OVHHTTPD20_ERRORLOG \[%{HTTPDERROR_DATE:timestamp}\] \[%{LOGLEVEL:loglevel}\] (?:\[client %{IPORHOST:clientip}\] ){0,1}%{GREEDYDATA:message}
OVHHTTPD24_ERRORLOG \[%{HTTPDERROR_DATE:timestamp}\] \[%{WORD:module}:%{LOGLEVEL:loglevel}\] \[pid %{POSINT:pid}(:tid %{NUMBER:tid})?\]( \(%{POSINT:proxy_errorcode}\)%{DATA:proxy_message}:)?( \[client %{IPORHOST:clientip}:%{POSINT:clientport}\])?( %{DATA:errorcode}:)? %{GREEDYDATA:message}
OVHHTTPD_ERRORLOG %{OVHHTTPD20_ERRORLOG}|%{OVHHTTPD24_ERRORLOG}

OVHHTTPD20_ERRORLOG_REFERRER \[%{HTTPDERROR_DATE:timestamp}\] \[%{LOGLEVEL:loglevel}\] (?:\[client %{IPORHOST:clientip}\] ){0,1}%{DATA:message}, referer: %{GREEDYDATA:referrer}
OVHHTTPD24_ERRORLOG_REFERRER \[%{HTTPDERROR_DATE:timestamp}\] \[%{WORD:module}:%{LOGLEVEL:loglevel}\] \[pid %{POSINT:pid}(:tid %{NUMBER:tid})?\]( \(%{POSINT:proxy_errorcode}\)%{DATA:proxy_message}:)?( \[client %{IPORHOST:clientip}:%{POSINT:clientport}\])?( %{DATA:errorcode}:)? %{DATA:message}, referer: %{GREEDYDATA:referrer}
OVHHTTPD_ERRORLOG_REFERRER %{OVHHTTPD20_ERRORLOG_REFERRER}|%{OVHHTTPD24_ERRORLOG_REFERRER}
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.