Lumberjack input vs beats input considerations (can I remove lumberjack intake altogether?)

The elastic doc states " Consider using the Beats input plugin instead. The Beats input implements the Lumberjack protocol v1 and v2."

However, there is insufficient information about how to make such a consideration and I am looking here to surface , from the community, some of the pros/cons of using lumberjack input vs beats input in logstash.

Today, I have a logstash pipeline with beats input and another with lumberjack input.

  1. We have beats output to the beats intake of logstash2 and integration tests ok.
  2. We have client logstash1 output to lumberjack intake on logstash2 ok but no integration tests yet.

Here are some approaches to not only adding a test for the logstash to logstash but changing the production to not directly even use a lumberjack intake.

2a. Introduce a client filebeat, for testing, output to logstash2 using the existing lumberjack intake. This means I don't have to stand up a client logstash to test the lumberjack intake. I tried this and it is not viable. I found the event delivered to lumberjack intake must have certain fields like "line","file","offset" set to certain values of expected type. It's a waste of time to essentially develop a logstash-forwarder with a beats configuration.

2b. Introduce a client logstash1, for testing, outputing via lumberjack to the the existing lumberjack intake of logstash2. This means I have to stand up a client logstash and I know it will work.

2c. Introduce a client logstash1, for testing, outputing via lumberjack to the the existing beats intake of logstash2. I am reluctant to do this if I have a known working lumberack intake; I see no benefits.

2d. Remove the logstash2 lumberjack intake from logstash2. The logstash1 is outputing lumberjack to logstash2 beats intake. For testing, stand up a logstash1 to exercise this.

2e. I know there is a possibility to even not even support logstash to logstash but go strictly with beats to logstash which might even be ideal but it impacts our application teams so I will defer that alternative for now.


This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.