# Enrichment with recent logs **URL:** \ **Category:** Logstash\ **Created:** [March 21, 2019, 10:36pm UTC](https://discuss.elastic.co/t/enrichment-with-recent-logs/173410 "2019-03-21T22:36:58Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![farsonic](https://avatars.discourse-cdn.com/v4/letter/f/46a35a/32.png) [@farsonic](https://discuss.elastic.co/u/farsonic)\ **Post date:** [March 21, 2019, 10:36pm UTC](https://discuss.elastic.co/t/enrichment-with-recent-logs/173410/1 "2019-03-21T22:36:58Z")
Hello, I have a logstash server receiving security events from BRO and also from a firewall. Upon ingest I've created a new log field that merges source-ip, source-port, dest-ip and dest-port into a single field called src\_dst. This field is present in both my BRO events and also in my firewall events. They look like this; src\_dst: 192.168.100.3\_49778\_23.12.57.18\_443 The BRO log enters the system prior to the firewall log, and the timestamp is usually half a second or so ahead of the first firewall entry. I'm attempting to have logstash lookup the src\_dst event from the firewall log and extract other fields (initially a single field called ja3) from the BRO logs. This is what I'm using for my elasticsearch filter. elasticsearch { hosts =\> ["X.X.X.X:9200"] index =\> "logstash-bro-%{+YYYY.MM.DD}" query\_template =\> "/etc/logstash/data/template.json" fields =\> { "ja3\_hash" =\> "ja3" } } ruby { code =\> 'event.set("ja3\_hash",event.get("ja3"))' } Here is my template.json file {"query": { "term": { "src\_dst": { "value": "%{[src\_dst]}"} } }, "\_source": ["ja3\_hash"] } When this is run I'm only getting a "-" value entered into the populated ja3\_hash field in the firewall event. This is the first time I've attempted to use the elasticsearch filter for enrichment, so possibly doing something incorrect here. F ---
**Author:** ![farsonic](https://avatars.discourse-cdn.com/v4/letter/f/46a35a/32.png) [@farsonic](https://discuss.elastic.co/u/farsonic)\ **Post date:** [March 22, 2019, 9:20pm UTC](https://discuss.elastic.co/t/enrichment-with-recent-logs/173410/2 "2019-03-22T21:20:15Z")
Just validated my search from the command line also. I'm getting results back from the CLI/Curl but always a null entry in my log events. curl -XPOST 'localhost:9200/\_search?pretty' -H 'Content-Type: application/json' -d ' { > "size": 1, > "query": { > "term": { > "src\_dst": { > "value": "192.168.100.201\_59481\_192.168.100.240\_9080"} > } > }, > "\_source": ["version", "ja3\_hash", "ja3s\_hash", "cipher"] > }' > { > "took" : 6, > "timed\_out" : false, > "\_shards" : { > "total" : 42, > "successful" : 42, > "skipped" : 0, > "failed" : 0 > }, > "hits" : { > "total" : 1, > "max\_score" : 3.2834144, > "hits" : [ > { > "\_index" : "logstash-bro-2019.03.81", > "\_type" : "doc", > "\_id" : "za4so2kBv9DUFyxXaAmG", > "\_score" : 3.2834144, > "\_source" : { > "cipher" : "TLS\_ECDHE\_RSA\_WITH\_AES\_256\_CBC\_SHA384", > "ja3s\_hash" : "9099266b09da09a1d9e1839ae9ad5682", > "ja3\_hash" : "decfb48a53789ebe081b88aabb58ee34", > "version" : "TLSv12" > } > } > ] > } > } ---
**Author:** ![farsonic](https://avatars.discourse-cdn.com/v4/letter/f/46a35a/32.png) [@farsonic](https://discuss.elastic.co/u/farsonic)\ **Post date:** [March 22, 2019, 11:21pm UTC](https://discuss.elastic.co/t/enrichment-with-recent-logs/173410/3 "2019-03-22T23:21:09Z")
ok, I've hardcoded my search now to not use a variable passed from my filter and this works. Looks like this filter can't lookup data from recent events? Is there any restrictions here that we need to be aware of? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 23, 2019, 5:48am UTC](https://discuss.elastic.co/t/enrichment-with-recent-logs/173410/4 "2019-03-23T05:48:59Z")
For an event to be searchable in Elasticsearch, it must first have passed all the way through the Logstash pipeline and been successfully written to Elasticsearch. As Logstash batches up events the full batch need to be successfully processed. Once it is in Elasticsearch it has to wait for a refresh to occur to be made searchable, which by default is initiated once per second. How long this process takes will depend on how much load the cluster is under, the latency of the bulk request and how long the refresh operation takes once initiated. This can likely take at least a few seconds, so if your events arrive close in time this type of solution might not work. You may want to have a look at [this blog post](https://www.elastic.co/blog/elasticsearch-data-enrichment-with-logstash-a-few-security-examples) which discussed enrichment and talks about a prototype memcached plugin that could perhaps be used to achieve much lower latencies and be suitable, although I have not tried it out. ---
**Author:** ![farsonic](https://avatars.discourse-cdn.com/v4/letter/f/46a35a/32.png) [@farsonic](https://discuss.elastic.co/u/farsonic)\ **Post date:** [March 23, 2019, 6:09am UTC](https://discuss.elastic.co/t/enrichment-with-recent-logs/173410/5 "2019-03-23T06:09:40Z")
Yep, I'm running my tests with subsequent packets and seems to be taking 10's of seconds to be searchable....the filter syntax is correct though. I'll look into this memchched plugin 🙂 Cheers F ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 20, 2019, 6:09am UTC](https://discuss.elastic.co/t/enrichment-with-recent-logs/173410/6 "2019-04-20T06:09:40Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.