Implementing transit codec for logstash

Elastic Stack Logstash
1

As per the title, I am looking for help writing a transit codec for logstash, any help would be much appreciated!

2

What is "transit"? Is it tied to a specific product?

3

transit is a codec.

Please refer to this forum post for more context:
https://groups.google.com/d/msg/logstash-users/yio-JfIWzCw/_4lCP6rvL6MJ

4

I see. From what I gather from that thread, you're looking for assistance in writing a codec using the transit ruby gem. Would that be accurate?

5

Yes you are correct, I currently have an implementation however, whenever I specify the --pluginpath to the path of the plugin file, it says it cannot find it, therefore I still haven't had a chance to test my code. Could it be because I am including the filename in the path? (i.e. should I exclude the file name) Also, if you have an idea of what the codec implementation should be, by all means please share! I would like to compare it against what I have if that is possible.

6

Am currently trying to figure out where to place the transit.rb file that I wrote, in order to have it successfully execute when specifying the pluginpath flag.

Also, I would like to know if this decode function in my codec is complete or am I missing something? The following is my codec implementation, please advise..

require "logstash/codecs/base"
require "logstash/codecs/line"

class LogStash::Codecs::Transit < LogStash::Codecs::Base

  # Setting the config_name here is required. This is how you
  # configure this filter from your logstash config.
  #
  # codec =>
  #   transit { ... }

  config_name "transit"

  # New plugins should start life at milestone 1.
  milestone 1

  # Replace the message with this value.
  config :message, :validate => :string

  public
  def register
    require "transit"
  end # def register

  public
  def decode(data)
    begin
      yield Transit::Reader.new(:json, data)
    rescue
      @logger.info("Transit parse failure. Falling back to plain-text", :error => e, :data => data)
      yield LogStash::Event.new("message" => data)
    end
  end # def decode

  public
  def encode(data)
    # unimplemented
  end # def encode
end # class LogStash::Filters::Transit
7

So I updated the codec, and would like to test it with logstash.

However, when I run logstash using:

/opt/logstash/bin/logstash agent --verbose --log /opt/logstash/log/logstash.logs --config /opt/logstash.conf --pluginpath /opt/logstash/lib/ --debug

This is the output that I receive:

Sending logstash logs to /opt/logstash/log/logstash.logs.
Using milestone 1 input plugin 'lumberjack'. This plugin should work, but would benefit from use by folks like you. Please let us know if you find bugs or have suggestions on how to improve this plugin.  For more information on plugin milestones, see http://logstash.net/docs/1.4.2/plugin-milestones {:level=>:warn}
Using milestone 2 input plugin 'file'. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.4.2/plugin-milestones {:level=>:warn}
Using milestone 1 codec plugin 'transit'. This plugin should work, but would benefit from use by folks like you. Please let us know if you find bugs or have suggestions on how to improve this plugin.  For more information on plugin milestones, see http://logstash.net/docs/1.4.2/plugin-milestones {:level=>:warn}
Error: Configuration problem.

Any ideas?

8

You're still using Logstash 1.4.2. I recommend following the instructions at http://www.elastic.co/guide/en/logstash/current/_how_to_write_a_logstash_codec_plugin.html and using Logstash 1.5.0-RC3 (RC4 should be out shortly).

We'll have a much easier time helping you with the new modular plugin system of 1.5, and you won't have to use --pluginpath.

As far as the output from 1.4 goes, nothing but the Error: Configuration problem. is unusual, and I can't tell off-hand why you got that.

9

This tool is being used in a prod environment, therefore I can't upgrade to RC version unfortunately...