Correlating data from 2 sources

[10past10 (Danny)]

Starting with the following;

• 1 large data-set of network traffic to be investigated (JSON)
• 1 tls-fingerprinting data-set or 1 'bad/blocked' IP data-set or ....

Use case;
When both the network traffic and fingerprint/ip data-sets are in Elasticsearch, how is it possible to correlate the values from both sources? Like some sort of translation, where the network-data gets matched with a fingerprinting data-set.

I would like to add a row to each event in the large data-set of network traffic, saying like 'Bad IP: Yes/No', so that I can create timelines or other visualizations only on data containing 'bad ip's'. Or matching specific values like a TLS fingerprint, adding an application name row to a network event based on a fingerprinting database.

For example using fingerprints found here: https://github.com/LeeBrotherston/tls-fingerprinting

Does anyone have any ideas about how to accomplish this with the Elastic stack?

[warkolm (Mark Walkom)]

That's a join, which ES/KB cannot do for you at this time.

The best option would be to use something like a translate filter in LS during ingestion and then do the lookup there.

[10past10 (Danny)]

Thanks for your fast reply! I wanted to know for sure that this option isn't in the Elastic-stick, before I start to look at other solutions.

I'm using bro to process the pcap's and create the json files. I think I either need to write some sort of python script or see if I can modify bro to create some extra output I want (something like what's described here: https://www.securityartwork.es/2017/02/02/tls-client-fingerprinting-with-bro/). I think the best option would be to add specific fields to the JSON events created with bro. Hmm interesting