Hello,
I try to get Sophos Firewall logs into logstash (which is working) but my grok filter to get the logline separated into fields is not working. The test via grok debugger is working...
Here are my logstash conf files:
Input (/etc/logstash/conf.d/02-sophos.conf):
input {
beats {
port => 5044
}
}
Filter (/etc/logstash/conf.d/03-sophos.conf):
filter {
grok {
break_on_match => true
match => ["message",'(%{MONTHNUM}):(?:%{MONTHDAY})-(?:%{HOUR}):(?:%{MINUTE}):(?:%{SECOND}) (?:%{SYSLOGHOST}) (?:%{SYSLOGPROG}): id=\"%{INT:utm_id}\" .* sub=\"%{DATA:utm_sub}\"']
match => ["message",'(%{MONTHNUM}):(?:%{MONTHDAY})-(?:%{HOUR}):(?:%{MINUTE}):(?:%{SECOND}) (?:%{SYSLOGHOST}) (?:%{SYSLOGPROG}): id=\"%{INT:utm_id}\"']
match => ["message",'(%{MONTHNUM}):(?:%{MONTHDAY})-(?:%{HOUR}):(?:%{MINUTE}):(?:%{SECOND}) (?:%{SYSLOGHOST}) (?:%{SYSLOGPROG}): \[%{DATA:utm_security2}:.*\]']
overwrite => ["MONTHNUM", "MONTHDAY", "HOUR", "MINUTE", "SECOND", "SYSLOGHOST", "SYSLOGPROG", "id"]
tag_on_failure => []
}
if [program] == "ulogd" {
if [utm_sub] == "packetfilter" {
grok {
match => ["message", '.* sys=\"%{DATA:utm_sys}\" sub=\"%{DATA}\" name=\"%{DATA:utm_name}\" action=\"%{DATA:utm_action}\" fwrule=\"%{INT:utm_fwrule}\" ?(initf=\"%{DATA:utm_initf}\")? ?(outitf=\"%{DATA:utm_outif}\")? ?(srcmac=\"%{MAC:utm_srcmac}\")? ?(dstmac=\"%{MAC:utm_dstmac}\")? srcip=\"%{IP:utm_srcip}\" dstip=\"%{IP:utm_dstip}\" proto=\"%{INT:utm_protocol}\" length=\"%{INT:utm_ulogd_pkglength}\" tos=\"%{DATA:utm_ulogd_tos}\" prec=\"%{DATA:utm_ulogd_prec}\" ttl=\"%{INT:utm_ttl}\" srcport=\"%{INT:utm_srcport}\" dstport=\"%{INT:utm_dstport}\" ?(tcpflags=\"%{DATA:utm_tcpflags}\")? ?(info=\"%{DATA:utm_info}\")?']
}
}
}
}
Output (/etc/logstash/conf.d/99-output.conf):
output {
elasticsearch {
hosts => ["http://localhost:9200"]
}
}
The pipeline gets loaded and there is no error in syslog for logstash:
Jun 1 10:47:29 MYHOST logstash[7904]: [2021-06-01T10:47:29,294][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/etc/logstash/conf.d/02-sophos.conf", "/etc/logstash/conf.d/03-sophos.conf", "/etc/logstash/conf.d/99-output.conf"], :thread=>"#<Thread:0x5f92f38f run>"}
Jun 1 11:00:24 MYHOSTis logstash[9416]: [2021-06-01T11:00:24,946][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>0.76}
Jun 1 11:00:24 MYHOSTis logstash[9416]: [2021-06-01T11:00:24,965][INFO ][logstash.inputs.beats ][main] Starting input listener {:address=>"0.0.0.0:5044"}
Jun 1 11:00:24 MYHOSTis logstash[9416]: [2021-06-01T11:00:24,978][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
Jun 1 11:00:25 MYHOSTis logstash[9416]: [2021-06-01T11:00:25,056][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
Jun 1 11:00:25 MYHOSTis logstash[9416]: [2021-06-01T11:00:25,064][INFO ][org.logstash.beats.Server][main][50bff7b0728f6691dc1728677b53f476bf52dd2357237c2832559249e7b4872b] Starting server on port: 5044
(MYHOSTis my anonymized elk stack hostname)
How can I debug this?
Regards
3lastic
