Use external file to enrich logs in Logstash

We have an enrichment file which holds data like this

device|IP|Site|Address|Country|Customer|<other properties>

We have a standard syslog file

Aug  1 14:30:52 _x.x.x.x_ 1628986: Aug  1 14:30:52.040 BST: %ILPOWER-5-IEEE_DISCONNECT: Interface Fa0/6: PD removed
Aug  1 14:30:52 _x.x.x.x_ 21996: Aug  1 14:30:51.070 BST: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/18, changed state to down
Aug  1 14:30:52 _x.x.x.x_ 21997: Aug  1 14:30:52.076 BST: %LINK-3-UPDOWN: Interface FastEthernet0/18, changed state to down

How can we match the IP address in the logs to the IP in the CSV, so that we can those extra files as properties?

dictionary plugin supports 1 key 1 value as per my understanding, i may be wrong.

Our current config file is below.

input
{
        file
        {
                type => "syslog"
                path => [ "/var/log/messages_syslog" ]
        start_position => "beginning"
        sincedb_path => "/dev/null"
        }
}
filter
{
        grok
        {
                match =>
                [   "message", "%{SYSLOGTIMESTAMP:timestamp1} %{IP:IP} %{NUMBER:bytes}: %{SYSLOGTIMESTAMP:timestamp2} %{WORD:Timezone}: %%{WORD:facility}-%{WORD:severit
y}-%{WORD:failure}: %{GREEDYDATA:Log-message}",
                        "message", "%{SYSLOGTIMESTAMP:timestamp1} %{IPORHOST:IP} %{WORD:Stack}: %{PROG:program}: \*%{GREEDYDATA:task}: %{SYSLOGTIMESTAMP:timestamp2}: \#
%{WORD:facility}-%{WORD:severity}-%{WORD:failure}: %{GREEDYDATA:Log-message}",
                                        "message", "%{SYSLOGTIMESTAMP:timestamp1} %{IPORHOST:IP} %{WORD:Stack}: %{PROG:program}: \*%{GREEDYDATA:task}: %{SYSLOGTIMESTAMP
:timestamp2}: %%{WORD:facility}-%{WORD:severity}-%{WORD:failure}: %{GREEDYDATA:Log-message}",
                                        "message", "%{SYSLOGTIMESTAMP:timestamp1} %{IPORHOST:IP} %{PROG:program}: \*%{GREEDYDATA:task}: %{SYSLOGTIMESTAMP:timestamp2}: %
%{WORD:facility}-%{WORD:severity}-%{WORD:failure}: %{GREEDYDATA:LogMessage}",
                                        "message", "%{SYSLOGTIMESTAMP:timestamp1} %{IP:IP} %{NUMBER:bytes}: \*%{SYSLOGTIMESTAMP:timestamp2} %{WORD:Timezone}: %%{WORD:fa
cility}-%{WORD:severity}-%{WORD:failure}: %{GREEDYDATA:Log-message}" ]
                                add_field => [ "received_at", "%{@timestamp}" ]
                                add_field => [ "received_from", "%{host}" ]
                                break_on_match => true
        }

        translate
        {
                field => "severity"
                destination => "severity_name"

                dictionary => [
                    "0", "Emergency",
                    "1", "Alert",
                    "2", "Critical",
                    "3", "Error",
                    "4", "Warning",
                    "5", "Notifications",
                    "6", "Information",
                    "7", "Debug"
                    ]
        }
}

output {
  stdout { codec => rubydebug }
    if "_grokparsefailure" not in [tags]
        {
                elasticsearch
                {
                        hosts => ["localhost:9200"] 
                        index  => "custsyslog"
                }
        }
}

Regards
Nayyar

dictionary plugin supports 1 key 1 value as per my understanding, i may be wrong.

That's right, but the value returned by the translate filter could be a JSON string that you pass to a json filter to expand into multiple fields.

1 Like

Could you suggest an example ?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.