Help with Logstash plug-in development

I am trying to develop a new logstash plugin in my development Elasticsearch stack environment. I am new to this so this is all a learning experience for me. I will try to be as clear and concise as I can be including as much information and code snippets as I can so here goes.

I started with generating the skeleton for my new plug-in using the logstash-plugin generate utility. Once I had this in place I then was able to build my plug-in using the gem build utility which ran without error and created a gem for me. I then ran the logstash-plugin install utility which successfully installed my plug-in.

The entry in my logstash.gem looks like this--
gem "logstash-output-s3-4.3.3", "0.1.0", :path => "./vendor/bundle/jruby/2.5.0/gems/logstash-output-s3-4.3.3"

I then took the code from the actual logstash plug-in that I'm hoping to modify and copied it into my new plug-in directory and then created a new logstash.conf file to test with using my new plug-in and started the logstash service. It ran as expected moving data from the servers that were being watched in the appropriate s3 buckets.

logstash.conf output section--

output {
s3{
region => "us-east-1"
bucket => "vertexsmb-osplogs-lower"
prefix => "Dev/Request-Logs/us-east-1/%{+YYYY}/%{+MM}/%{+dd}/"
server_side_encryption => true
server_side_encryption_algorithm => "aws:kms"
ssekms_key_id => "arn:aws:kms:us-east-1:409148389496:alias/LowerEnvironmentS3"
validate_credentials_on_root_bucket => false
codec => "json_lines"
size_file => 1024000
time_file => 5
rotation_strategy => "size_and_time"
temporary_directory => "../../LogstashS3OutputData/temp/"
canned_acl => "private"
}

s3{
	region => "us-east-1"
	bucket => "vertexsmb-osplogs-lower"
	prefix => "Dev/Response-Logs/us-east-1/%{+YYYY}/%{+MM}/%{+dd}/"
	server_side_encryption => true
	server_side_encryption_algorithm => "aws:kms"
	ssekms_key_id => "arn:aws:kms:us-east-1:409148389496:alias/LowerEnvironmentS3"
	validate_credentials_on_root_bucket => false
	codec => "json_lines"
	size_file => 1024000
	time_file => 5
	rotation_strategy => "size_and_time"
	temporary_directory => "../../LogstashS3OutputData/temp/"
	canned_acl => "private"

}
}

My next step was and this is where I am having issues was to try and create a new property for the plug-in called file_prefix--

s3{
	region => "us-east-1"
	bucket => "vertexsmb-osplogs-lower"
	prefix => "Dev/Request-Logs/us-east-1/%{+YYYY}/%{+MM}/%{+dd}/"

=> file_prefix => "test"
server_side_encryption => true
server_side_encryption_algorithm => "aws:kms"
ssekms_key_id => "arn:aws:kms:us-east-1:409148389496:alias/LowerEnvironmentS3"
validate_credentials_on_root_bucket => false
codec => "json_lines"
size_file => 1024000
time_file => 5
rotation_strategy => "size_and_time"
temporary_directory => "../../LogstashS3OutputData/temp/"
canned_acl => "private"
}

I modified the following files in my solution
\vendor\bundle\jruby\2.5.0\gems\logstash-output-s3-4.3.3\spec\outputs\s3_spec.rb

  • added "file_prefix" => file_prefix at line 15

\vendor\bundle\jruby\2.5.0\gems\logstash-output-s3-4.3.3\spec\supports\helpers.rb

  • added let(:file_prefix) {} at line line 12
  • added "file_prefix" => file_prefix, at line 19

\vendor\bundle\jruby\2.5.0\gems\logstash-output-s3-4.3.3\lib\logstash\outputs\s3.rb

  • added config :file_prefix, :validate => :string right after the statement config :prefix, :validate => :string, :default => '' at line 154
  • added @file_prefix = file_prefix at line 226

Now when I start the logstash service I get this error message
Unknown setting 'file_prefix' for s3

I've searched and I've tried everything I can think but I can't get past this hurdle.

TIA,
Bill Youngman

This can be considered closed as I am using the Feature Request #215 ( Added filename as a configuration option.) for the s3 Output plugin as the starting point for my project and so far haven't run into any problems.

-b

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.