# Use executable as filter

**URL:** <https://discuss.elastic.co/t/use-executable-as-filter/76489>\
**Category:** Logstash\
**Created:** [February 25, 2017, 5:14pm UTC](https://discuss.elastic.co/t/use-executable-as-filter/76489 "2017-02-25T17:14:33Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![benaryorg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benaryorg/32/15854_2.png) [@benaryorg](https://discuss.elastic.co/u/benaryorg)\
**Post date:** [February 25, 2017, 5:14pm UTC](https://discuss.elastic.co/t/use-executable-as-filter/76489/1 "2017-02-25T17:14:33Z")

</div>

Hi,

as I'm just getting started with Logstash, ELK and Beats (Filebeats to be exact) I'm trying to comprehend the vast amount of information thrown at me.  
The whole stack seems to adhere to the [Unix Philosophy](https://en.wikipedia.org/wiki/Unix_philosophy), which I think is great and made learning straight forward.

Every single piece of the stack does exactly one thing.  
The relevant parts for my problem are only Filebeats and Logstash.

Filebeats seems very much like a tool meant to **only** transfer the logs from a node to the next logstash instance.  
It has some really rough filtering options but I assume they are only meant for things that really shouldn't ever leave the local machine, e.g. Debug loglevel.

Then there is Logstash which does filtering, parsing and the like.  
Now, I do have a Unix-ish executable.  
I run it, I put my logs into stdin and I get them on stdout as JSON objects, one on each line.  
Having a unique tool is a great pleasure, as maintenance and debugging both get way easier as the infrastructure is already present and it doesn't take any knowledge to get the first ten lines of a logfile in JSON (`head file | executable`).

How do I use that executable and where does it belong?  
I've seen Filebeat already supports parsing a JSON string from a field, but really I wouldn't want that as Beats is essentially some transfer tool for logs that prevents duplicates, losses and all the things that happen with raw copying.  
It's also kinda real-time.

Logstash on the other hand seems the perfect place to do that with all it's filters, but there isn't a single one that allows a program to be executed as a child process of Logstash with all the good Unix-ish stdio stuff.  
I could of course write my own filter but that seems kind of weird for a software with such an amount of official plugins.  
And just using the _ruby_ filter with `exec()` really doesn't cut it as I do not want to spin up the process on every single message.  
I mean, what if I had some state which I needed to keep?  
Everything else would need buffering, `select()`s and whatnot.

I also don't want to use the _input-pipe_ as I had to build quite some stuff around my simple executable to make it talk to Beats all on it's own.

This just doesn't seem right, am I missing something?  
Is there an easy way to do that (_grok_ is **not** a solution, neither are _fifo_s or other sockets)?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 25, 2017, 5:14pm UTC](https://discuss.elastic.co/t/use-executable-as-filter/76489/2 "2017-03-25T17:14:39Z")

</div>

Closed on March 25, 2017, 5:14pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
