Logstash | Apache Acces log | Parsing

Elastic Stack Logstash
1

Hi All,

I have custom apache log format, I am struggling to make it parse using logstash. Can any one help me.

LogFormat "%{X-Forwarded-For}i %h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i" "%{True-Client-IP}i" %q"

Sample Log output;

Actual Client = 2.50.172.5
2.20.249.8, 2.20.133.107 = distil IP
172.31.24.155 = Load Blancer IP

Feb 21 05:24:43 ip-172-31-24-246 apache[29290]: 2.50.172.5 , 2.20.249.8, 2.20.133.107 172.31.24.155 - - [21/Feb/2017:05:24:39 +0000] "GET /ar/property/get_category_trends/?property_id=2439117 HTTP/1.1" 200 401 "https://www.example.com/ar/to-rent/apartments/abu-dhabi/corniche-area/saraya/live-your-way-2-b-r-apartment-w-city-view-2439117.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" "2.50.172.5" ?property_id=2439117

2

HI @aliahsan81 ,

did you try %{COMBINEDAPACHELOG:log} ?

and take a look at:

grok debugger
grok patterns

also did a small example how it could work:

%{SYSLOGTIMESTAMP:Timestamp}\s%{DATA}\W\s%{IP:Actual_Client}\s\W\s%{IP:distil_IP}\W\s%{IP:distil_IP}\s%{IP:Load_Blancer_IP}\s\W\s\W\s\W%{HTTPDATE:HttpDate}\W\s%{QUOTEDSTRING:request}\s%{INT:Http_code}\s%{INT:bytes}\s%{QUOTEDSTRING:request2}\s%{QUOTEDSTRING:client_info}\s%{QUOTEDSTRING:client_ip_request}\s%{DATA:property}$

named the fields with the names you provided, maybe that helps :wink:

3

Thanks

I have added your advice but I think its not working please have a look, Output of logstash

{
"path" => "/mnt/efs/all_logs_httpd/apache.log",
"@timestamp" => 2017-02-21T11:59:14.135Z,
"geoip" => {},
"@version" => "1",
"host" => "ip-172-31-26-77",
"message" => "Feb 19 03:14:11 ip-172-31-24-246 apache[13865]: 77.66.11.145, 23.65.29.108, 195.10.11.229 - - [19/Feb/2017:03:14:10 +0000] "GET /to-rent/apartments/dubai/dubai-marina/mag-218-tower/spacious-2br-with-golf-course-view-in-dubai-marina-2359643.html HTTP/1.1" 410 28082 "-" "uipbot/1.0 (uipbot@semasio.net)"",
"type" => "apache-access",
"tags" => [
[0] "_grokparsefailure",
[1] "_geoip_lookup_failure"
]
}

Please havea look at my logstash.conf

input {

file {
path => "/mnt/efs/all_logs_httpd/apache.log"
type => "apache-access"
start_position => "beginning"
}
}

filter {
if [type] == "apache-access" { # this is where we use the type from the input section
grok {
match => [ "message", "%{SYSLOGTIMESTAMP:Timestamp}\s%{DATA}\W\s%{IP:Actual_Client}\s\W\s%{IP:distil_IP}\W\s%{IP:distil_IP}\s%{IP:Load_Blancer_IP}\s\W\s\W\s\W%{HTTPDATE:HttpDate}\W\s%{QUOTEDSTRING:request}\s%{INT:Http_code}\s%{INT:bytes}\s%{QUOTEDSTRING:request2}\s%{QUOTEDSTRING:client_info}\s%{QUOTEDSTRING:client_ip_request}\s%{DATA:property}$" ]
}

}

geoip {
source => "clientip"
}
}

output {
elasticsearch {
#user => "elastic"

password => "dsasa"

hosts => localhost

index => "example.com-%{+YYYY.MM.dd}"
}
stdout {
codec => rubydebug
}
}

"tags" => [
[0] "_grokparsefailure",
[1] "_geoip_lookup_failure"
]
}

4

Hi @aliahsan81 ,

the above filter dont work because there are some differences compared to the first log.

try using only %{COMBINEDAPACHELOG:apache_log}

grok {
match => [ "message", "%{COMBINEDAPACHELOG:apache_log}" ]
}

5

Hi @lueneburger

Thanks for advise, I used grok debugger grok patterns and able to prase sample request using grok debugger. But when I inseart same regex in logstash it did not break break my apache log (message) part. Can you please advise what will be the issue.

Ali