Trying to implement an IP Watch List in SIEM and wondering if there was any guidance/best practices in doing so.
I currently have an index where new malicious IP objects are logged on a regular basis, let's say these objects contain a field named malicious.srcip
I'd like to detect when logs in other indexes, like packetbeat, have a destination IP address (we'll call the packetbeat field dstip) that matches ANY malicious.srcip in my malicious IP index.
Is there any already-known wisdom on how to accomplish such a thing using Kibana/SIEM? I'm thinking of trying to use Watcher to do this, but I'm not sure how to structure the query so that it will look across all the objects in my malicious IP index as the "value" for the filter. Thoughts?
I had implemented a similar use case by using a threat intel feed. The workflow was as given below:
Used logstash blueliv plugin (available on github) to download threat feeds to an ES index by creating a pipeline for the same.
After indexing enrichment was done for incoming network events
For enrichment 3 components were required: enrich policy, enrich processor and ingest pipeline
First, enrichment policy was created which had the source index and the field of interest in the threat-intel index along with fields that will be added after enrichment. (Source index here was the threat-intel index and enrich fields were fields like reputation score etc.)
Second, execute the enrich policy to create enrich index which will be finally used for enrichment. (POST /_enrich/policy/!enrich-policy!/_execute) this will create a hidden index .enrich which is only created when you run this execute API. The enrich index never updates instead a new one is created and old one is deleted every time policy is executed.
Third , Create an ingest pipeline to do the enrichment in a logstash pipeline(or ingest node). Ingest pipeline has enrich policy, field of target index to be compared with enrich index source field for enrichment and target field which will be enriched field indexed on ES.
After this setup the only thing that's required is calling the ingest pipeline in a logstash pipeline for enrichment.
Malicious events can be queried by checking existence of new fields added by ingest pipeline.
Worked perfectly well for me, just took me enough time to figure this out. Hope it helps you and saves time as well.
The approach of keeping threat intel in ES and checking against it for every ingested log entry is quite heavy, in my opinion. For sites with significant throughput this can create unneeded burden on ES (you would end up with more nodes). With smaller sites you would have to deploy logstash nodes, while otherwise a lightweight shipper would be just fine. Instead, I would recommend to look for a traffic analysis tool with pluggable threat intel module. It could be Zeek, for example, instead of packetbeat. Then you can put filebeat with zeek module, or 3rd party ingest pipeline on the zeek host.
Thanks for idea, while I'm fairly new to elastic still exploring different ways to do it. Makes sense to do it on a lightweight shipper instead.
would try this out!
Do you have approach for a huge deployment? I have almost 500GB/day ingestion that requires threat-intel lookup. Logstash is ingesting the data using syslog inputs and 5ES nodes in HA.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.