Hi All,
I'm new to logstash / grok and I'm getting errors with my grok statement that I was hoping somebody could help me with?
My log entry looks like this:
50.60.70.80 [08/Aug/2018:16:25:48 +0100] 50.60.70.72 /Common/Application_HTTPS/VirtualServer /Common/Application_HTTPS/Service_Pool 50.60.70.87 \"\" \"\" 13897 6365 \"\" \"\"
And this is my conf file. The first grok statement is what should be handling this traffic:
filter{
### GROK Statement to catch Traffic from *** Application ###
if [type] == "syslog" {
if "Application_HTTPS" in [message] {
grok {
match => { "message" => "%{URIHOST:client_ip} %{SYSLOG5424SD:timestamp} %{IP:virtual_ip} %{URIPATHPARAM:virtual_name} %{URIPATHPARAM:virtual_pool_name} %{IP:server} %{NUMBER:server_port} \\\"\\\" \\\"\\\" %{NUMBER:packet_size:bytes:int} %{NUMBER:response:ms:int} \\\"\\\" \\\"\\\""}}
translate {
regex => true
dictionary_path => "/etc/logstash/jsontranslate.yml"
field => "message"
}
json {
source => "translation"
}
} else {
grok {
match => { "message" => "%{IP:clientip} \[%{HTTPDATE:timestamp}\] %{IP:virtual_ip} %{DATA:virtual_name} %{DATA:virtual_pool_name} %{DATA:server} %{NUMBER:server_port} \"%{DATA:path}\" \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response:int} %{NUMBER:bytes:int} %{NUMBER:response_ms:int} %{QS:referrer} %{QS:agent}"}}
### TRANSLATE statement to convert incoming IP Addresses to Geo-Location using jsontranslate.yml ###
translate {
regex => true
dictionary_path => "/etc/logstash/jsontranslate.yml"
field => "message"
}
json {
source => "translation"
}
}
}
}
output {
elasticsearch {
hosts => ["10.128.10.10:9200"]
index => "logstash-%{+YYYY.MM.dd}"
}
stdout { codec => rubydebug}
}
I've tested this on https://grokdebug.herokuapp.com/ and it works well but not in logstash.
Message:
50.60.70.80 [08/Aug/2018:16:25:48 +0100] 50.60.70.72 /Common/Application/VirtualServer /Common/Application/Service_Pool 50.60.70.87 443 \"\" \"\" 13897 6365 \"\" \"\"
Pattern:
%{URIHOST:client_ip} %{SYSLOG5424SD:timestamp} %{IP:virtual_ip} %{URIPATHPARAM:virtual_name} %{URIPATHPARAM:virtual_pool_name} %{IP:server} %{NUMBER:server_port} \\\"\\\" \\\"\\\" %{NUMBER:packet_size:bytes:int} %{NUMBER:response:ms:int} \\\"\\\" \\\"\\\"
Response:
{
"client_ip": [
[
"50.60.70.80"
]
],
"IPORHOST": [
[
"50.60.70.80"
]
],
"HOSTNAME": [
[
"50.60.70.80"
]
],
"IP": [
[
null
]
],
"IPV6": [
[
null,
null,
null
]
],
"IPV4": [
[
null,
"50.60.70.72",
"50.60.70.87"
]
],
"port": [
[
null
]
],
"timestamp": [
[
"[08/Aug/2018:16:25:48 +0100]"
]
],
"DATA": [
[
"08/Aug/2018:16:25:48 +0100"
]
],
"virtual_ip": [
[
"50.60.70.72"
]
],
"virtual_name": [
[
"/Common/Application/VirtualServer"
]
],
"URIPATH": [
[
"/Common/Application/VirtualServer",
"/Common/Application/Service_Pool"
]
],
"URIPARAM": [
[
null,
null
]
],
"virtual_pool_name": [
[
"/Common/Application/Service_Pool"
]
],
"server": [
[
"50.60.70.87"
]
],
"server_port": [
[
"443"
]
],
"BASE10NUM": [
[
"443",
"13897",
"6365"
]
],
"packet_size": [
[
"13897"
]
],
"response": [
[
"6365"
]
]
}