Can Logstash Interact with a C Library?

[redapplesonly (RAO)]

Hi Logstash Ninja Masters,

I’m using an instance of Logstash (Docker container version 7.4.0, yes, I know I need to upgrade) that is currently processing our network data. Here’s a sample data record:

“10.10.10.10”, “172.217.9.4”, “TCP”, “80”, “12345”, “2000”

Meaning “Host 10.10.10.10 sent 2,000 bytes to host 172.217.9.4 over TCP ports 80 and 12345.” Simple, networking-101 type stuff.

But what I’d like to do is translate all of this into more meaningful data. nDPI is a C library which can translate the above data and turn it into:

“10.10.10.10”, “HTTP”, “www.google.com”, “2000”

…which is much more meaningful.

Unfortunately, the nDPI library looks like it is only written in C. The question is, how can Logstash interact with a C library?

I’m tempted to write a standalone C program that could query the nDPI library on a per-need basis. Then, perhaps Logstash could query my C program through a network call, or something? I’m not sure what is the best approach. Another consideration: nDPI would return a struct containing lots of information, so I’d have to devise a way to transfer all that information out of my C program and back into Logstash.

But before I invest what would probably be significant development time, I’d like to ask the forum: is the best way? Can someone recommend any other approach? Any advice will be appreciated.

[Badger]

If you want to do a reverse lookup of an IP address then you can use a dns filter. If you want to map TCP port 80 to "HTTP" then a translate filter that loads something similar to /etc/services should be able to do it.

That said, if you want to call a C library from a ruby filter I see no reason why that would not work. As I have said before, if you want to use logstash as a C++ compiler all you need is a very long ruby filter.

[redapplesonly (RAO)]

Thanks Badger, appreciate the options and insightful feedback. Logstash is new to me, so I have a lot to learn.

I think the solution is to use the C libraries within Ruby code, and then run that Ruby code inside a Ruby filter. I'll research from there, but no doubt be back on the forums with detailed, follow-up questions. Thanks again, I really love this forum.