Hi,
Im having an issue with logsasher, after sending couple of events it stop sending logs to elastic. basically Im inputing AWS ELB logs, extracting the url_path on first match and continue extracting additional data from the "url_path"
I`ve set costume patterns:
DEVICEBREAK [^/]+[^/]+/(?<api_version>[^/]+)+/devices/%{WORD:request_type:String}[?]device_id=%{WORD:device_id}&key=%{WORD:key}
KEYBREAK [^/]+[^/]+/(?<api_version>[^/]+)+/devices/%{WORD:request_type:String}[?]key=%{WORD:key}&device_id=%{WORD:device_id}
APIBREAK [^/]+[^/]+/(?<api_version>[^/]+)+/api_wrappers/connection/domain[?]key=%{WORD:key}&domain_api_secret=%{WORD:api_secret}&user_id=%{BASE10NUM:user_id}
SPIDRBREAK [^/]+[^/]+/(?<api_version>[^/]+)+/users/%{WORD:request_type:String}/communications/spidr[?]key=%{WORD:key}
USERBREAK [^/]+[^/]+/(?<api_version>[^/]+)+/users/devices[?]key=%{WORD:key}
DATEBREAK [^/]+[^/]+/(?<api_version>[^/]+)+/users/%{WORD:request_type:String}/data_channel[?]key=%{WORD:key}
GSSBREAK [^/]+[^/]+/(?<api_version>[^/]+)+/gss/wrapper/user[?]key=%{WORD:key}
CREDBREAK [^/]+[^/]+/(?<api_version>[^/]+)+/devices/%{WORD:request_type:String}/push[?]key=%{WORD:key}&device_id=%{WORD:device_id}
My config file:
input {
file {
path => "/var/log/s3/elb-us-logs/*.log"
type => "elb"
start_position => "beginning"
sincedb_path => "log_sincedb"
}
}
filter {
if [type] == "elb" {
grok {
break_on_match => false
patterns_dir => "./patterns"
match => [ "message" , [
"%{NOTSPACE:loadbalancer} %{IP:client_ip}:%{NUMBER:client_port:int} %{IP:backend_ip}:%{NUMBER:backend_port:int} %{NUMBER:request_processing_time:float} %{NUMBER:backend_processing_time:float} %{NUMBER:response_processing_time:float} %{NUMBER:elb_status_code:int} %{NUMBER:backend_status_code:int} %{NUMBER:received_bytes:int} %{NUMBER:sent_bytes:int} "%{WORD:method:string} %{URI:url_path} HTTP%{URIPATHPARAM:httpversion}"",
"%{NOTSPACE:loadbalancer} %{IPORHOST:clientIp}:%{NUMBER:backend_port:int} - -1 -1 -1 %{NUMBER:elb_status_code:int}"
]
]
patterns_dir => "./patterns"
match => [ "url_path", [
"%{DEVICEBREAK}",
"%{KEYBREAK}",
"%{APIBREAK}",
"%{SPIDRBREAK}",
"%{USERBREAK}",
"%{DATEBREAK}",
"%{GSSBREAK}",
"%{CREDBREAK}"
]
]
}
}
The log file looks like this:
"2015-05-25T09:45:51.110114Z extelb-name-API 100.0.54.51:56841 172.18.3.80:80 0.000049 0.023448 0.000022 200 200 0 57 "GET https://api.somedomain.io:443/v1.1/devices/messages?device_id=7A7509DE74DF4654B0919878D7C84B3D&key=UAT4fa560d246c8476ba27a3dda3a64d630 HTTP/1.1"",
Any help will be appreciated, thanks.