# Use external file to enrich logs in Logstash

**URL:** <https://discuss.elastic.co/t/use-external-file-to-enrich-logs-in-logstash/144836>\
**Category:** Logstash\
**Created:** [August 17, 2018, 7:35am UTC](https://discuss.elastic.co/t/use-external-file-to-enrich-logs-in-logstash/144836 "2018-08-17T07:35:00Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Nayyar\_Coder](https://avatars.discourse-cdn.com/v4/letter/n/c2a13f/32.png) [@Nayyar\_Coder](https://discuss.elastic.co/u/Nayyar_Coder)\
**Post date:** [August 17, 2018, 7:35am UTC](https://discuss.elastic.co/t/use-external-file-to-enrich-logs-in-logstash/144836/1 "2018-08-17T07:35:00Z")

</div>

We have an enrichment file which holds data like this

```
device|IP|Site|Address|Country|Customer|<other properties>

```

We have a standard syslog file

```
Aug 1 14:30:52 _x.x.x.x_ 1628986: Aug 1 14:30:52.040 BST: %ILPOWER-5-IEEE_DISCONNECT: Interface Fa0/6: PD removed
Aug 1 14:30:52 _x.x.x.x_ 21996: Aug 1 14:30:51.070 BST: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/18, changed state to down
Aug 1 14:30:52 _x.x.x.x_ 21997: Aug 1 14:30:52.076 BST: %LINK-3-UPDOWN: Interface FastEthernet0/18, changed state to down

```

How can we match the IP address in the logs to the IP in the CSV, so that we can those extra files as properties?

dictionary plugin supports 1 key 1 value as per my understanding, i may be wrong.

Our current config file is below.

```
input
{
        file
        {
                type => "syslog"
                path => ["/var/log/messages_syslog"]
        start_position => "beginning"
        sincedb_path => "/dev/null"
        }
}
filter
{
        grok
        {
                match =>
                [ "message", "%{SYSLOGTIMESTAMP:timestamp1} %{IP:IP} %{NUMBER:bytes}: %{SYSLOGTIMESTAMP:timestamp2} %{WORD:Timezone}: %%{WORD:facility}-%{WORD:severit
y}-%{WORD:failure}: %{GREEDYDATA:Log-message}",
                        "message", "%{SYSLOGTIMESTAMP:timestamp1} %{IPORHOST:IP} %{WORD:Stack}: %{PROG:program}: \*%{GREEDYDATA:task}: %{SYSLOGTIMESTAMP:timestamp2}: \#
%{WORD:facility}-%{WORD:severity}-%{WORD:failure}: %{GREEDYDATA:Log-message}",
                                        "message", "%{SYSLOGTIMESTAMP:timestamp1} %{IPORHOST:IP} %{WORD:Stack}: %{PROG:program}: \*%{GREEDYDATA:task}: %{SYSLOGTIMESTAMP
:timestamp2}: %%{WORD:facility}-%{WORD:severity}-%{WORD:failure}: %{GREEDYDATA:Log-message}",
                                        "message", "%{SYSLOGTIMESTAMP:timestamp1} %{IPORHOST:IP} %{PROG:program}: \*%{GREEDYDATA:task}: %{SYSLOGTIMESTAMP:timestamp2}: %
%{WORD:facility}-%{WORD:severity}-%{WORD:failure}: %{GREEDYDATA:LogMessage}",
                                        "message", "%{SYSLOGTIMESTAMP:timestamp1} %{IP:IP} %{NUMBER:bytes}: \*%{SYSLOGTIMESTAMP:timestamp2} %{WORD:Timezone}: %%{WORD:fa
cility}-%{WORD:severity}-%{WORD:failure}: %{GREEDYDATA:Log-message}" ]
                                add_field => ["received_at", "%{@timestamp}"]
                                add_field => ["received_from", "%{host}"]
                                break_on_match => true
        }

        translate
        {
                field => "severity"
                destination => "severity_name"

                dictionary => [
                    "0", "Emergency",
                    "1", "Alert",
                    "2", "Critical",
                    "3", "Error",
                    "4", "Warning",
                    "5", "Notifications",
                    "6", "Information",
                    "7", "Debug"
                    ]
        }
}

output {
  stdout { codec => rubydebug }
    if "_grokparsefailure" not in [tags]
        {
                elasticsearch
                {
                        hosts => ["localhost:9200"] 
                        index => "custsyslog"
                }
        }
}

```

Regards  
Nayyar

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 17, 2018, 7:44am UTC](https://discuss.elastic.co/t/use-external-file-to-enrich-logs-in-logstash/144836/2 "2018-08-17T07:44:05Z")

</div>

> dictionary plugin supports 1 key 1 value as per my understanding, i may be wrong.

That's right, but the value returned by the translate filter could be a JSON string that you pass to a json filter to expand into multiple fields.

---

<div class="post-metadata">

**Author:** ![Nayyar\_Coder](https://avatars.discourse-cdn.com/v4/letter/n/c2a13f/32.png) [@Nayyar\_Coder](https://discuss.elastic.co/u/Nayyar_Coder)\
**Post date:** [August 17, 2018, 7:45am UTC](https://discuss.elastic.co/t/use-external-file-to-enrich-logs-in-logstash/144836/3 "2018-08-17T07:45:02Z")

</div>

Could you suggest an example ?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 14, 2018, 7:50am UTC](https://discuss.elastic.co/t/use-external-file-to-enrich-logs-in-logstash/144836/4 "2018-09-14T07:50:02Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.