# Near Realtime Threat Intel enrichment using custom external sources stored on enrichment indexes **URL:** \ **Category:** Logstash\ **Created:** [November 18, 2021, 4:46am UTC](https://discuss.elastic.co/t/near-realtime-threat-intel-enrichment-using-custom-external-sources-stored-on-enrichment-indexes/289525 "2021-11-18T04:46:22Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![claudio.rifo](https://avatars.discourse-cdn.com/v4/letter/c/aeb1de/32.png) [@claudio.rifo](https://discuss.elastic.co/u/claudio.rifo)\ **Post date:** [November 18, 2021, 4:46am UTC](https://discuss.elastic.co/t/near-realtime-threat-intel-enrichment-using-custom-external-sources-stored-on-enrichment-indexes/289525/1 "2021-11-18T04:46:22Z")
Hi There. First Ill try to explain my general idea (that I already have implemented and is working) then the problems I have and a few questions. The general idea is to have a Logstash ingest pipeline, lets call it Pipeline A (with any kind of source that includes an IP). This Pipeline queries a Elasticsearch Threat intel Index for information on the IP, then: A – If there is information, then Logstash outputs the event to an Elasticsearch Ingest Pipeline that enriches the data and store it. B – If there is no information: B.1 - the pipeline generates a new event and outputs the new event to another Logstash PipeLine, Lets call it Pipeline B. B.2 – the pipeline outputs the original event to an Elasticsearch Ingest Pipeline, that on this case will have no information to enrich the data, then is stored. Pipeline B receives the generated event with the IP (pipeline to pipeline communication). Then it queries external services for information on the IP. And then the recovered information from external services its posted on the Threat intel Index. Some of my problems and questions are: - Enrichment indexes require to be “Updated” to acknowledge new data posted on the Threat Intel Index (POST /\_enrich/policy//\_execute). - Current model create events with no threat intel (only when the IP is seen for a first time). So I’m considering to send non enrich data (Step B.2) to a temporal index, then execute a re-index to an ingest pipeline that stores enriched data into final index (Same as B.1). I know that I could avoid a lot of problems if instead of doing data enrichment on Elasticsearch I just do it on Logstash. But I think that having this kind of setup simplifies the solution a lot when working with multiple kinds of incoming data sources (avoids using a collector pattern). Finally. What do you think of the logical construct of the model? Do you see any issues or have any comments on how to Improve it? ---
**Author:** ![claudio.rifo](https://avatars.discourse-cdn.com/v4/letter/c/aeb1de/32.png) [@claudio.rifo](https://discuss.elastic.co/u/claudio.rifo)\ **Post date:** [November 25, 2021, 3:05am UTC](https://discuss.elastic.co/t/near-realtime-threat-intel-enrichment-using-custom-external-sources-stored-on-enrichment-indexes/289525/2 "2021-11-25T03:05:13Z")
After further testing I have found another issue that I have no idea how to avoid. Basically, when multiple events from the same IP arrive really fast (for the first time) all events get forwarded to pipeline B, and this generates multiple queries to External service (consuming API Requests, and adding additional unnecessary lag to the process) and also creates duplicated events into the Threat intel Index. I know that I could avoid duplicates on the Threat intel Index by using the IP filed as document ID. But the extra http requests seems unavoidable. The only thing that comes to my mind is to avoid using the generated event for external search. And replace that pipeline with a recurrent search (deduplicated query) on the temporal index for the IPs that have no Threat Intel. Any ideas? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 23, 2021, 3:05am UTC](https://discuss.elastic.co/t/near-realtime-threat-intel-enrichment-using-custom-external-sources-stored-on-enrichment-indexes/289525/3 "2021-12-23T03:05:17Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.