Hi Everyone,
I am very new to Elastic Stack so I have very less knowledge regarding Logstash grok filter. Basically I am trying to get the logs with priority number so that syslog_pri {} plugin will generate all the necessary fields in ES. Firewall logs already contains priority code and its easy but I am struggling with application logs like kafka, nifi and hadoop which comes with no pri and I don`t know if its even possible as the logs are different. I know its not upto logastash to generate pri code on logs but is it possible or do anyone know how to do that in application ? Currently using version 7.3.1 version logstash, ES and kibana same across. As an example I am listing the logs I am getting from all the components.
NIFI LOG
2020-01-14 11:34:48,769 INFO [NiFi Web Server-91] org.apache.nifi.web.filter.RequestLogger Attempting request for (anonymous) GET http://10.X.X.X:9999/nifi-api/flow/cluster/summary (source ip: 10.X.X.X)
HADOOP LOG
2020-01-15 10:26:07,021 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Cache Size Before Clean: 0, Total Deleted: 0, Public Deleted: 0, Private Deleted: 0
KAFKA LOG
2020-01-15 10:25:33,106] DEBUG [Controller id=0] Topics not in preferred replica for broker 0 Map() (kafka.controller.KafkaController)
FIREWALL LOG with pri <189>
<189>date=2020-01-15 time=10:10:20 devname="FIREWALL389" devid="FIREWALL389" logid="0001000014" type="traffic" subtype="local" level="notice" vd="root" eventtime=1579083020 srcip=fe80::f602:70ff:fe9f:e9aa srcport=5353 srcintf="mgmt" srcintfrole="lan" dstip=ff02::fb dstport=5353 dstintf=unknown-0 dstintfrole="undefined" sessionid=338794 proto=17 action="deny" policyid=0 policytype="local-in-policy6" service="udp/5353" dstcountry="Reserved" srccountry="Reserved" trandisp="noop" app="udp/5353" duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 rcvdpkt=0 appcat="unscanned"
Logstash config for firewall:
input {
udp {
port => 1514
type => "syslog"
}
}
filter {
if [type] == "syslog" {
grok {
match => { "message" => ["<%{NUMBER:syslog_pri}>%{GREEDYDATA:Log-message}"] }
}
syslog_pri {}
}
}
output {
elasticsearch {
hosts => "10.X.X.X:9200"
index => "firewall"
user => elastic
password => password
document_type => "syslogs"
}
}
GROK filter is generating all these fields from firewall logs with pri <189>
syslog_facility: local7
syslog_facility_code: 23
syslog_pri: 189
syslog_severity: notice
syslog_severity_code: 5
type: syslog