Hi,
as I'm just getting started with Logstash, ELK and Beats (Filebeats to be exact) I'm trying to comprehend the vast amount of information thrown at me.
The whole stack seems to adhere to the Unix Philosophy, which I think is great and made learning straight forward.
Every single piece of the stack does exactly one thing.
The relevant parts for my problem are only Filebeats and Logstash.
Filebeats seems very much like a tool meant to only transfer the logs from a node to the next logstash instance.
It has some really rough filtering options but I assume they are only meant for things that really shouldn't ever leave the local machine, e.g. Debug loglevel.
Then there is Logstash which does filtering, parsing and the like.
Now, I do have a Unix-ish executable.
I run it, I put my logs into stdin and I get them on stdout as JSON objects, one on each line.
Having a unique tool is a great pleasure, as maintenance and debugging both get way easier as the infrastructure is already present and it doesn't take any knowledge to get the first ten lines of a logfile in JSON (head file | executable).
How do I use that executable and where does it belong?
I've seen Filebeat already supports parsing a JSON string from a field, but really I wouldn't want that as Beats is essentially some transfer tool for logs that prevents duplicates, losses and all the things that happen with raw copying.
It's also kinda real-time.
Logstash on the other hand seems the perfect place to do that with all it's filters, but there isn't a single one that allows a program to be executed as a child process of Logstash with all the good Unix-ish stdio stuff.
I could of course write my own filter but that seems kind of weird for a software with such an amount of official plugins.
And just using the ruby filter with exec() really doesn't cut it as I do not want to spin up the process on every single message.
I mean, what if I had some state which I needed to keep?
Everything else would need buffering, select()s and whatnot.
I also don't want to use the input-pipe as I had to build quite some stuff around my simple executable to make it talk to Beats all on it's own.
This just doesn't seem right, am I missing something?
Is there an easy way to do that (grok is not a solution, neither are fifos or other sockets)?