Parse WebLogic Access Log File

Elastic Stack Logstash
1

Hey guys,

i am trying to parse the WebLogic access log file with the pattern below, but it doesn't match.

Log Example

2015-11-30 18:45:54 0.001 1164 GET /sample-app/ 404 - "1.0A3sN0eF000000000;kXjE" - -
2015-11-30 18:45:54 0.099 11190 GET /sample-app/index.html 200 "8i01F1T_000000000" "1.8i01F1T_000000000;kXjE" - -
2015-11-30 18:45:54 0.074 11190 GET /sample-app/index.html 200 "JnBRZ0Qj000000000" "1.JnBRZ0Qj000000000;kXjE" - -
2015-11-30 18:45:54 0.097 11190 GET /sample-app/index.html 200 "ZWqCq0cF000000000" "1.ZWqCq0cF000000000;kXjE" - -
2015-11-30 18:45:54 0.09 11190 GET /sample-app/index.html 200 "cK0NU1bC000000000" "1.cK0NU1bC000000000;kXjE" - -
2015-11-30 18:45:54 0.004 11190 GET /sample-app/index.html 200 "3Y_BW0VC000000000" "1.3Y_BW0VC000000000;kXjE" - -
2015-11-30 18:45:54 0.002 11190 GET /sample-app/index.html 200 "fKHGR0aF000000000" "1.fKHGR0aF000000000;kXjE" - -
2015-11-30 18:45:54 0.002 11190 GET /sample-app/index.html 200 "Ld4Ua0kF000000000" "1.Ld4Ua0kF000000000;kXjE" - -

This are the fields explanation
date time time-taken bytes cs-method cs-uri sc-status sc(X-ORACLE-DMS-ECID) cs(ECID-Context) cs(Proxy-Remote-User) cs(Proxy-Client-IP)

This was the pattern that i used

%{ISO8601_DATE:date}%{TIME:time}%{NUMBER:time_taken:float}%{IPORHOST:c_ip}(?:%{IPORHOST:x_ClientIP}|-)%{NUMBER:sc_status:int}(?:%{NUMBER:bytes:int}|-)(?:-|%{USERNAME:x_AuthUser})%{WORD:cs_method}%{NOTSPACE:cs_uri}%{DATA:x_UserAgent}(?:-|%{DATA:x_Referer})%{DATA:x_Scheme}%{DATA:x_Protocol}%{GREEDYDATA:x_AcceptLanguage}

The log is being tab separated. Can anyone please help me?

Logstash version: 2.0
Elasticsearch version: 2.0

2

It's been a long while since I worked on WebLogic, but I believe you can alter the files at the Log4j level, which is easier than using grok. At the very least, it would allow you to use custom separators so you could more easily tokenize with either the grok or kv filters.

3

Thanks Aaron, yes i can change it at Log4j level, but this will be used for several domains, then i will need to update a lot of weblogic servers, so at this point i am trying to parse the original log file. This is problably why i am not able to parse this access logs.

4

There doesn't appear to be any spaces in your grok expression in which case that's one obvious reason why things aren't matching.

To debug grok expressions I recommend starting off simple with e.g.

%{ISO8601_DATE:date}.*

and making sure that works. Pipe test data to Logstash via a stdin input and use a stdout { codec => rubydebug } output (or use https://github.com/magnusbaeck/logstash-filter-verifier). Then add one token at a time, i.e. your next step would be

%{ISO8601_DATE:date} %{TIME:time}

and eventually you'll zero in on the problem.

5

I tried to use this pattern but i got an error.

root@vagrant-ubuntu-trusty-64:/opt/logstash# bin/logstash -f /etc/logstash/conf.d/1.conf
Default settings used: Filter workers: 1
The error reported is:
pattern %{ISO8601_DATE:date} not defined

my conf file

input {
stdin {
type => "stdin-type"
}
}

filter {

grok {
match => {
message => ["%{ISO8601_DATE:date} %{TIME:time} %{NUMBER:time_taken:float}%{IPORHOST:c_ip}(?:%{IPORHOST:x_ClientIP}|-)%{NUMBER:sc_status:int}(?:%{NUMBER:bytes:int}|-)(?:-|%{USERNAME:x_AuthUser})%{WORD:cs_method}%{NOTSPACE:cs_uri}%{DATA:x_UserAgent}(?:-|%{DATA:x_Referer})%{DATA:x_Scheme}%{DATA:x_Protocol}%{GREEDYDATA:x_AcceptLanguage}"]
}
}

}

output {
stdout { codec => rubydebug }
}

6

The error reported is:
pattern %{ISO8601_DATE:date} not defined

There is no grok pattern named ISO8601_DATE. Where did you get the idea to use it? Perhaps you need to configure Logstash to use additional pattern files. But TIMESTAMP_ISO8601 exists and should match your timestamp.

7

I solved the problem with the pattern above

"%{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day}[\t]%{HOUR:hour}:?%{MINUTE:minute}(?::?%{SECOND:second})?[\t]%{NUMBER:time_taken:float}[\t]%{NUMBER:bytes}[\t]%{WORD:cs_method}[\t]%{NOTSPACE:cs_uri}[\t]%{NUMBER:response:int}[\t]%{QUOTEDSTRING:dmsecid}[\t]%{QUOTEDSTRING:ecidcontext}[\t]%{IPORHOST:xip}[\t]%{IPORHOST:clientip}"]

Thanks guys