Parse failure Apache accesslog with empty remote user

[RomainD (Romain)]

Hello,

I have a _grokparsefailure when the remote user is empty, let's show you the configurations files:

Apache v2.4

Logstash v5.6.3

apache2.conf, my custom log format, %u is the remote user :

LogFormat "%a %l %u [%{%F}t %{%T}t.%{msec_frac}t] \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined_cid

A sample log with parse failure, "" is the problem, third parameter :

10.10.10.10 - "" [2017-11-16 09:51:52.099] "GET /user/1609/groups HTTP/1.0" 200 1622 "https://services.com/browse" "Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0"

And in logstash filters :

HTTPD_COMBINEDLOG_WITHMICROSEC %{IPORHOST:clientip} %{HTTPDUSER:ident} %{HTTPDUSER:auth} \[%{APP_DATETIME:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent}

My conclusion is %u of Apache is setting "" as value in log, but the original HTTPD_COMBINEDLOG uses %{HTTPDUSER:auth} described as below :

HTTPDUSER %{EMAILADDRESS}|%{USER}
EMAILADDRESS %{EMAILLOCALPART}@%{HOSTNAME}
USER %{USERNAME}
USERNAME [a-zA-Z0-9._-]+

My solution is to allow " in the username pattern like:

USERNAME [a-zA-Z0-9._-"]+

or modify %{HTTPDUSER:auth} in the HTTPD_COMBINEDLOG pattern like

(%{HTTPDUSER:auth}|"")

or modify something in the core of apache, or put a remote user in all clients calls by default but with thousands of production servers...

Thank you for your advices.

[guyboertje]

Seems to me that you will still want the auth field and its value should be an empty string.

If so then modify the HTTPD_COMBINEDLOG pattern:

(%{HTTPDUSER:auth}|"%{\s?:auth}")

or similar

[RomainD (Romain)]

Yes I want the remote user field provided by apache with %u.

Doc about apache LogFormat : https://httpd.apache.org/docs/current/fr/mod/mod_log_config.html

But if apache sets "" by default if the remote user is empty, the HTTPD_COMBINEDLOG of grok should handle it by default in my opinion.

Or I have a custom apache configuration introducing a bug and having "" for remote user if empty isn't a standard situation. ?

[guyboertje]

All the apache log examples I can find use a - dash if the remote_user is unknown.

http://www.monitorware.com/en/logsamples/apache.php
http://ossec-docs.readthedocs.io/en/latest/log_samples/apache/apache.html
https://kb.webtrends.com/articles/Information/Sample-Log-File-Apache-Extended-Log-File-Format
https://raw.githubusercontent.com/aagea/elk-example/master/apache_logs

[RomainD (Romain)]

Okay, I found that this value "" is posted by our SSO Shibboleth when we are using non protected urls.
For our Apache servers it's a configuration in the Shibboleth SP.

So we need to implement a custom grok pattern.

Sorry for the inconvenience, thank you for your answers.