We have an enrichment file which holds data like this
device|IP|Site|Address|Country|Customer|<other properties>
We have a standard syslog file
Aug 1 14:30:52 _x.x.x.x_ 1628986: Aug 1 14:30:52.040 BST: %ILPOWER-5-IEEE_DISCONNECT: Interface Fa0/6: PD removed
Aug 1 14:30:52 _x.x.x.x_ 21996: Aug 1 14:30:51.070 BST: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/18, changed state to down
Aug 1 14:30:52 _x.x.x.x_ 21997: Aug 1 14:30:52.076 BST: %LINK-3-UPDOWN: Interface FastEthernet0/18, changed state to down
How can we match the IP address in the logs to the IP in the CSV, so that we can those extra files as properties?
dictionary plugin supports 1 key 1 value as per my understanding, i may be wrong.
Our current config file is below.
input
{
file
{
type => "syslog"
path => [ "/var/log/messages_syslog" ]
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter
{
grok
{
match =>
[ "message", "%{SYSLOGTIMESTAMP:timestamp1} %{IP:IP} %{NUMBER:bytes}: %{SYSLOGTIMESTAMP:timestamp2} %{WORD:Timezone}: %%{WORD:facility}-%{WORD:severit
y}-%{WORD:failure}: %{GREEDYDATA:Log-message}",
"message", "%{SYSLOGTIMESTAMP:timestamp1} %{IPORHOST:IP} %{WORD:Stack}: %{PROG:program}: \*%{GREEDYDATA:task}: %{SYSLOGTIMESTAMP:timestamp2}: \#
%{WORD:facility}-%{WORD:severity}-%{WORD:failure}: %{GREEDYDATA:Log-message}",
"message", "%{SYSLOGTIMESTAMP:timestamp1} %{IPORHOST:IP} %{WORD:Stack}: %{PROG:program}: \*%{GREEDYDATA:task}: %{SYSLOGTIMESTAMP
:timestamp2}: %%{WORD:facility}-%{WORD:severity}-%{WORD:failure}: %{GREEDYDATA:Log-message}",
"message", "%{SYSLOGTIMESTAMP:timestamp1} %{IPORHOST:IP} %{PROG:program}: \*%{GREEDYDATA:task}: %{SYSLOGTIMESTAMP:timestamp2}: %
%{WORD:facility}-%{WORD:severity}-%{WORD:failure}: %{GREEDYDATA:LogMessage}",
"message", "%{SYSLOGTIMESTAMP:timestamp1} %{IP:IP} %{NUMBER:bytes}: \*%{SYSLOGTIMESTAMP:timestamp2} %{WORD:Timezone}: %%{WORD:fa
cility}-%{WORD:severity}-%{WORD:failure}: %{GREEDYDATA:Log-message}" ]
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
break_on_match => true
}
translate
{
field => "severity"
destination => "severity_name"
dictionary => [
"0", "Emergency",
"1", "Alert",
"2", "Critical",
"3", "Error",
"4", "Warning",
"5", "Notifications",
"6", "Information",
"7", "Debug"
]
}
}
output {
stdout { codec => rubydebug }
if "_grokparsefailure" not in [tags]
{
elasticsearch
{
hosts => ["localhost:9200"]
index => "custsyslog"
}
}
}
Regards
Nayyar