Logstash | Apache Acces log | Parsing

Hi All,

I have custom apache log format, I am struggling to make it parse using logstash. Can any one help me.

LogFormat "%{X-Forwarded-For}i %h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i" "%{True-Client-IP}i" %q"

Sample Log output;

Actual Client = 2.50.172.5
2.20.249.8, 2.20.133.107 = distil IP
172.31.24.155 = Load Blancer IP

Feb 21 05:24:43 ip-172-31-24-246 apache[29290]: 2.50.172.5 , 2.20.249.8, 2.20.133.107 172.31.24.155 - - [21/Feb/2017:05:24:39 +0000] "GET /ar/property/get_category_trends/?property_id=2439117 HTTP/1.1" 200 401 "https://www.example.com/ar/to-rent/apartments/abu-dhabi/corniche-area/saraya/live-your-way-2-b-r-apartment-w-city-view-2439117.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" "2.50.172.5" ?property_id=2439117

HI @aliahsan81 ,

did you try %{COMBINEDAPACHELOG:log} ?

and take a look at:

grok debugger
grok patterns

also did a small example how it could work:

%{SYSLOGTIMESTAMP:Timestamp}\s%{DATA}\W\s%{IP:Actual_Client}\s\W\s%{IP:distil_IP}\W\s%{IP:distil_IP}\s%{IP:Load_Blancer_IP}\s\W\s\W\s\W%{HTTPDATE:HttpDate}\W\s%{QUOTEDSTRING:request}\s%{INT:Http_code}\s%{INT:bytes}\s%{QUOTEDSTRING:request2}\s%{QUOTEDSTRING:client_info}\s%{QUOTEDSTRING:client_ip_request}\s%{DATA:property}$

named the fields with the names you provided, maybe that helps :wink:

Thanks

I have added your advice but I think its not working please have a look, Output of logstash

{
"path" => "/mnt/efs/all_logs_httpd/apache.log",
"@timestamp" => 2017-02-21T11:59:14.135Z,
"geoip" => {},
"@version" => "1",
"host" => "ip-172-31-26-77",
"message" => "Feb 19 03:14:11 ip-172-31-24-246 apache[13865]: 77.66.11.145, 23.65.29.108, 195.10.11.229 - - [19/Feb/2017:03:14:10 +0000] "GET /to-rent/apartments/dubai/dubai-marina/mag-218-tower/spacious-2br-with-golf-course-view-in-dubai-marina-2359643.html HTTP/1.1" 410 28082 "-" "uipbot/1.0 (uipbot@semasio.net)"",
"type" => "apache-access",
"tags" => [
[0] "_grokparsefailure",
[1] "_geoip_lookup_failure"
]
}

Please havea look at my logstash.conf

input {

file {
path => "/mnt/efs/all_logs_httpd/apache.log"
type => "apache-access"
start_position => "beginning"
}
}

filter {
if [type] == "apache-access" { # this is where we use the type from the input section
grok {
match => [ "message", "%{SYSLOGTIMESTAMP:Timestamp}\s%{DATA}\W\s%{IP:Actual_Client}\s\W\s%{IP:distil_IP}\W\s%{IP:distil_IP}\s%{IP:Load_Blancer_IP}\s\W\s\W\s\W%{HTTPDATE:HttpDate}\W\s%{QUOTEDSTRING:request}\s%{INT:Http_code}\s%{INT:bytes}\s%{QUOTEDSTRING:request2}\s%{QUOTEDSTRING:client_info}\s%{QUOTEDSTRING:client_ip_request}\s%{DATA:property}$" ]
}

}

geoip {
source => "clientip"
}
}

output {
elasticsearch {
#user => "elastic"

password => "dsasa"

hosts => localhost

index => "example.com-%{+YYYY.MM.dd}"
}
stdout {
codec => rubydebug
}
}

"tags" => [
[0] "_grokparsefailure",
[1] "_geoip_lookup_failure"
]
}

Hi @aliahsan81 ,

the above filter dont work because there are some differences compared to the first log.

try using only %{COMBINEDAPACHELOG:apache_log}

grok {
match => [ "message", "%{COMBINEDAPACHELOG:apache_log}" ]
}

Hi @lueneburger

Thanks for advise, I used grok debugger grok patterns and able to prase sample request using grok debugger. But when I inseart same regex in logstash it did not break break my apache log (message) part. Can you please advise what will be the issue.

Ali

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.