Enable golang for your logstash pipe
This is not a rewrite its an extensible idea. Has anyone come across this type idea before?
I have used Elasticstack for quite some time now. I can tell you unequivocally that the most time is spent on debug and understanding of logstash. It's old and large and very difficult to configure and debug. I envisioned something in Go with much less baggage. The fact that pipelines are headless makes debugging difficult so any degree of simplification or familiarity is helpful. For someone versed in go this may be attractive.
I had difficulty in my first pass at this writing a direct replacement to logstash. This was due to what seems to be a propriety of interface on the socket-ed input side. They used a layer (Lumberjack) on top of TCPIP which is not well described anywhere I could find. So instead of going to the logstash and/or filebeat code and reverse engineering something to work I had another idea. If I left a pass through logstash pipe in place that would solve that problem for me. Then the output starts a pipe to run egopipe. You write your filter stage in egopipe, recompile it and place the executable in logstash directory.
So I buried the first version and started to work on the new idea. The basic code fell together nicely in a few sessions over days. I debugged a handful of problems and immediately came up with a working model I can grow new feature.
The docs pass through logstash which instead of writing to an elastic index launches my pipe. This in effect allows us to use go to manipulate the doc as a map and then output it to elastic. I recently added ssl and authentication. If you have an existing elastic pipeline configured, calling egopipe is simple. Even with ssl enabled configuration is a breeze.
I need ...
-
feedback
-
ideas
-
testing
-
suggested feature/change
code is up on github at GitHub - wshekrota/egopipe: A minimalist solution for logstash complexity in Elastic Star me there if you like the idea.