Two copy of one log

erion · June 28, 2017, 3:46pm

Hi, when i upload a string of apache log in kibana i see one with real timestamp and the same string with timestamp of uploaded . I use the mapping founded on github here https://github.com/elastic/examples/tree/master/ElasticStack_apache.

This is json of timestamp uploaded.

This is json for real date timestamp. I use different string.
Thanks for replay

magnusbaeck · June 28, 2017, 5:51pm

In the first example your grok filter isn't working on your log line so the date filter that otherwise would take the timestamp from the log and store it in the @timestamp field doesn't work either.

erion · June 29, 2017, 7:51am

Hi @magnusbaeck i use:
filter {
grok {
match => {
"message" => '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} [%{HTTPDATE:timestamp}] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:respon$
}
}
date {
match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ]
target => "@timestamp"
locale => en
}
Now my question is i have the same type of log(apache), why grok filter match only some strings and not all strings? What is wrong?
Thank you for replay.

magnusbaeck · June 29, 2017, 8:41am

Show an example of a log line that the grok filter couldn't parse. Use copy/paste, don't post a screenshot. Please also post exactly what your grok filter looks like. The configuration you posted above is damaged (your grok expression hardly ends with "%{NUMBER:respon$"). Make sure you post your configuration as preformatted text using the </> toolbar button.

erion · June 29, 2017, 10:10am

@magnusbaeck , this is my grok config:

grok {
match => {
"message" => '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} [%{HTTPDATE:timestamp}] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBE:response:int} (?:-|%{NUMBER:bytes:int}) %{QS:referrer} %{QS:agent}'
}
}

I know you ask me a complete string of log but when i investigate of this issues i've founded some similar anomalies.
Watch yellow circle.

This is the correct match.
Now watch the damaged match.

For me the error is that my filter grok doesn't match the message started with "-"
Thanks for replays

magnusbaeck · June 29, 2017, 10:18am

Okay, so replace

%{IPORHOST:clientip}

with

(?:-|%{IPORHOST:clientip})

to make the initial IP address optional.

erion · June 29, 2017, 10:24am

Perfect, thank you very mutch @magnusbaeck

erion · June 30, 2017, 9:10am

Hi @magnusbaeck I have another problem with that filter grok and i think it's for request or HTTP verbs.

I don't know why but this json data don't be matched by filter grok.
Thanks for replay

magnusbaeck · June 30, 2017, 9:18am

Show an example of a log line that the grok filter couldn't parse. Use copy/paste, don't post a screenshot. Please also post exactly what your grok filter looks like.

erion · June 30, 2017, 9:28am

This is my grok filter

grok {
match => {
"message" => '(?:-|%{IPORHOST:clientip}) %{USER:ident} %{USER:auth} [%{HTTPDATE:timestamp}] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int}) %{QS:referrer} %{QS:agent}'
}
}

and this is two apache log string

"188.135.227 - - [14/Mar/2017:11:11:15 +0100] "BASELINE-CONTROL /banner/fileadmin/fileadmin/T2uCB7.htm HTTP/1.1" 200 33611 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)" "10.0.148861205248" "-" "59194""

"188.135.227 - - [14/Mar/2017:11:11:11 +0100] "X-MS-ENUMATTS /banner/fileadmin/vLkgrYq.htm HTTP/1.1" 200 33611 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)" "10.0.148861205248" "-" "61982""

Thank you very much @magnusbaeck for your help

magnusbaeck · June 30, 2017, 10:21am

Try %{NOTSPACE:verb} instead of %{WORD:verb}.

erion · June 30, 2017, 1:28pm

It go, but why word doesn't go?
Thank you very much for replay

magnusbaeck · June 30, 2017, 1:57pm

Because of the hyphens. \w doesn't include hyphens.

github.com

logstash-plugins/logstash-patterns-core/blob/v2.0.5/patterns/grok-patterns#L14


EMAILADDRESS %{EMAILLOCALPART}@%{HOSTNAME}
HTTPDUSER %{EMAILADDRESS}|%{USER}
INT (?:[+-]?(?:[0-9]+))
BASE10NUM (?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))
NUMBER (?:%{BASE10NUM})
BASE16NUM (?<![0-9A-Fa-f])(?:[+-]?(?:0x)?(?:[0-9A-Fa-f]+))
BASE16FLOAT \b(?<![0-9A-Fa-f.])(?:[+-]?(?:0x)?(?:(?:[0-9A-Fa-f]+(?:\.[0-9A-Fa-f]*)?)|(?:\.[0-9A-Fa-f]+)))\b


POSINT \b(?:[1-9][0-9]*)\b
NONNEGINT \b(?:[0-9]+)\b
WORD \b\w+\b
NOTSPACE \S+
SPACE \s*
DATA .*?
GREEDYDATA .*
QUOTEDSTRING (?>(?<!\\)(?>"(?>\\.|[^\\"]+)+"|""|(?>'(?>\\.|[^\\']+)+')|''|(?>`(?>\\.|[^\\`]+)+`)|``))
UUID [A-Fa-f0-9]{8}-(?:[A-Fa-f0-9]{4}-){3}[A-Fa-f0-9]{12}


# Networking
MAC (?:%{CISCOMAC}|%{WINDOWSMAC}|%{COMMONMAC})
CISCOMAC (?:(?:[A-Fa-f0-9]{4}\.){2}[A-Fa-f0-9]{4})

system · July 28, 2017, 1:57pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.