Splunk lookup file equivalent

Elastic Stack Elasticsearch
1

Hello Elastic Forum members,

I have been exploring the different possible data enrichments that are available in Splunk. However, I am trying to understand if there is a plugin or module that can be added to Elastic to create and manage lookup files, like the lookup editor in Splunk, that comes with Elastic. I also want to know if there is a way to put a lookup file into Elastic so that it can be used in data enrichment and searches. It will be needed for creating alerting.

Are there other alternatives to store this lookup data in other tools, like a database, and link it with Elastic via JDBC?

For example:

type,source,ip
C2, Test1, 192.168.1.13
C2,"Test1,192.168.1.14
TOR,Test2,192.168.2.23
TOR,Test2,192.168.2.24
Malware,Test1,192.168.1.69

I know I can put a single list of items and use a TERMS query clause. However, I have not yet finished experimenting to see if it supports wildcards and/or nest fields in the document to allow for more structured data.

PUT iplist/_doc/black2
   { 
        "ip": ["192.168.1.13","192.168.1.14","192.168.2.23","192.168.2.24","192.168.1.69"]
   }

GET iplist/_search

PUT traffic/_doc/_bulk
    { "index":{}}
    {"seq":1, "sip" : "192.168.8.2", "dip" : "192.168.1.13"}
    { "index":{}}
    {"seq":2, "sip" : "192.168.8.4", "dip" : "192.168.1.14"}
    { "index":{}}
    {"seq":3, "sip" : "192.168.8.8", "dip" : "192.168.2.23"}
    { "index":{}}
    {"seq":4, "sip" : "192.168.8.8", "dip" : "192.168.2.23"}
    { "index":{}}
    {"seq":5, "sip" : "192.168.8.2", "dip" : "192.168.1.13"}
    { "index":{}}
    {"seq":6, "sip" : "192.168.8.4", "dip" : "192.168.1.14"}
    { "index":{}}
    {"seq":7, "sip" : "192.168.1.69", "dip" : "192.168.8.2"}
    { "index":{}}
    {"seq":8, "sip" : "192.168.1.69", "dip" : "192.168.8.2"}

GET traffic/_search

GET traffic/_search?pretty
    {
        "query": {
        "terms": {
            "dip": {
                "index":"iplist",
                "id": "black2",
                "path":"ip"
                }
            }
        }
    }

Best Regards,
Weng Seng.

2

When working with the Elastic Stack this is generally done at index time. Please see this blog post for a more detailed discussion.

3

The ability to build/maintain dynamic whitelists and blacklists is not entirely covered by the schema-on-write or schema-on-read discussion. Even if the whitelists and/or blacklists are processed at ingestion, instead of search time, there needs to be a way to build and maintain them. Splunk has a way to maintain lookup files out of the box. I would like to know if there is an equivalent in Elastic.

The current course material I have taken does not cover logstash or ingestion pipelines. I will try to find more course material on ingestion pipelines to see if they can be used to generate alerts, or if alerting is still based on search time as implied by this document.

Do you have any examples of how ingestion pipelines are used to process firewall logs for Threat Intel hits, including retrospective searches covering events from the past. Do you have examples of how whitelists and blacklists might be implemented for IDS alerts? Where some alerts might be dynamically whitelisted for specific combinations of source/destination/signature id?
My attempts to find these examples via Google have not been successful.

I guess I am not the only person pondering this question, since Yelp has written ElastAlert to " * “Match when a certain field matches a blacklist/whitelist” ( blacklist and whitelist type)".

(ElastAlert - Easy & Flexible Alerting With Elasticsearch — ElastAlert 0.0.1 documentation)

Thanks,
Weng Seng.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.