COMMONAPACHELOG vs Explicit replacement

jmota · April 17, 2019, 2:38pm

gi,
can some1 help with this one ? ( ES 7, LS 7)

if i use COMMONAPACHELOG logstash works ...
if i replace it for its "long version" gives me an error...
( i only use 1 at a time, cant edit the coments)

full pattern taken from
#https://github.com/elastic/logstash/blob/v1.4.2/patterns/grok-patterns

grok { 
	match => { 
		"message" => "%{COMMONAPACHELOG} %{NUMBER:timetaken:int}"
		"message" => "%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{NUMBER:timetaken:int}"
	} 
}

error is
[2019-04-17T15:36:34,298][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.0.0"}
[2019-04-17T15:36:35,986][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:apache, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, {, } at line 11, column 93 (byte 269) after filter{\n\tgrok { \n\t\tmatch => { \n#\t\t\t"message" => "%{COMMONAPACHELOG} %{NUMBER:timetaken:int}"\n\t\t\t"message" => "%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "", :backtrace=>["C:/Kibana_ElasticSearch/logstash/logstash-core/lib/logstash/compiler.rb:41:in compile_imperative'", "C:/Kibana_ElasticSearch/logstash/logstash-core/lib/logstash/compiler.rb:49:incompile_graph'", "C:/Kibana_ElasticSearch/logstash/logstash-core/lib/logstash/compiler.rb:11:in block in compile_sources'", "org/jruby/RubyArray.java:2577:inmap'", "C:/Kibana_ElasticSearch/logstash/logstash-core/lib/logstash/compiler.rb:10:in compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:151:ininitialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in initialize'", "C:/Kibana_ElasticSearch/logstash/logstash-core/lib/logstash/java_pipeline.rb:23:ininitialize'", "C:/Kibana_ElasticSearch/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in execute'", "C:/Kibana_ElasticSearch/logstash/logstash-core/lib/logstash/agent.rb:325:inblock in converge_state'"]}

any toughts?

JM

jmota · April 17, 2019, 3:03pm

from

github.com

logstash-plugins/logstash-patterns-core/blob/master/patterns/httpd

HTTPDUSER %{EMAILADDRESS}|%{USER}
HTTPDERROR_DATE %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR}

# Log formats
HTTPD_COMMONLOG %{IPORHOST:clientip} %{HTTPDUSER:ident} %{HTTPDUSER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)
HTTPD_COMBINEDLOG %{HTTPD_COMMONLOG} %{QS:referrer} %{QS:agent}

# Error logs
HTTPD20_ERRORLOG \[%{HTTPDERROR_DATE:timestamp}\] \[%{LOGLEVEL:loglevel}\] (?:\[client %{IPORHOST:clientip}\] ){0,1}%{GREEDYDATA:message}
HTTPD24_ERRORLOG \[%{HTTPDERROR_DATE:timestamp}\] \[%{WORD:module}:%{LOGLEVEL:loglevel}\] \[pid %{POSINT:pid}(:tid %{NUMBER:tid})?\]( \(%{POSINT:proxy_errorcode}\)%{DATA:proxy_message}:)?( \[client %{IPORHOST:clientip}:%{POSINT:clientport}\])?( %{DATA:errorcode}:)? %{GREEDYDATA:message}
HTTPD_ERRORLOG %{HTTPD20_ERRORLOG}|%{HTTPD24_ERRORLOG}

# Deprecated
COMMONAPACHELOG %{HTTPD_COMMONLOG}
COMBINEDAPACHELOG %{HTTPD_COMBINEDLOG}

tryed with
"message" => "%{IPORHOST:clientip} %{HTTPDUSER:ident} %{HTTPDUSER:auth} [%{HTTPDATE:timestamp}] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{NUMBER:timetaken:int}"

same problem...

Badger · April 17, 2019, 3:09pm

If you have double quotes in your pattern then surround it using single quotes

match => {  "message" => '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{NUMBER:timetaken:int}' }

jmota · April 17, 2019, 3:41pm

Had to be something like tat !
1/2 a day lost and own sanity in check and it was a devilish detail.

Thanks a lot !

system · May 15, 2019, 3:45pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.