I am trying to develop a new logstash plugin in my development Elasticsearch stack environment. I am new to this so this is all a learning experience for me. I will try to be as clear and concise as I can be including as much information and code snippets as I can so here goes.
I started with generating the skeleton for my new plug-in using the logstash-plugin generate utility. Once I had this in place I then was able to build my plug-in using the gem build utility which ran without error and created a gem for me. I then ran the logstash-plugin install utility which successfully installed my plug-in.
The entry in my logstash.gem looks like this--
gem "logstash-output-s3-4.3.3", "0.1.0", :path => "./vendor/bundle/jruby/2.5.0/gems/logstash-output-s3-4.3.3"
I then took the code from the actual logstash plug-in that I'm hoping to modify and copied it into my new plug-in directory and then created a new logstash.conf file to test with using my new plug-in and started the logstash service. It ran as expected moving data from the servers that were being watched in the appropriate s3 buckets.
logstash.conf output section--
output {
s3{
region => "us-east-1"
bucket => "vertexsmb-osplogs-lower"
prefix => "Dev/Request-Logs/us-east-1/%{+YYYY}/%{+MM}/%{+dd}/"
server_side_encryption => true
server_side_encryption_algorithm => "aws:kms"
ssekms_key_id => "arn:aws:kms:us-east-1:409148389496:alias/LowerEnvironmentS3"
validate_credentials_on_root_bucket => false
codec => "json_lines"
size_file => 1024000
time_file => 5
rotation_strategy => "size_and_time"
temporary_directory => "../../LogstashS3OutputData/temp/"
canned_acl => "private"
}
s3{
region => "us-east-1"
bucket => "vertexsmb-osplogs-lower"
prefix => "Dev/Response-Logs/us-east-1/%{+YYYY}/%{+MM}/%{+dd}/"
server_side_encryption => true
server_side_encryption_algorithm => "aws:kms"
ssekms_key_id => "arn:aws:kms:us-east-1:409148389496:alias/LowerEnvironmentS3"
validate_credentials_on_root_bucket => false
codec => "json_lines"
size_file => 1024000
time_file => 5
rotation_strategy => "size_and_time"
temporary_directory => "../../LogstashS3OutputData/temp/"
canned_acl => "private"
}
}
My next step was and this is where I am having issues was to try and create a new property for the plug-in called file_prefix--
s3{
region => "us-east-1"
bucket => "vertexsmb-osplogs-lower"
prefix => "Dev/Request-Logs/us-east-1/%{+YYYY}/%{+MM}/%{+dd}/"
=> file_prefix => "test"
server_side_encryption => true
server_side_encryption_algorithm => "aws:kms"
ssekms_key_id => "arn:aws:kms:us-east-1:409148389496:alias/LowerEnvironmentS3"
validate_credentials_on_root_bucket => false
codec => "json_lines"
size_file => 1024000
time_file => 5
rotation_strategy => "size_and_time"
temporary_directory => "../../LogstashS3OutputData/temp/"
canned_acl => "private"
}
I modified the following files in my solution
\vendor\bundle\jruby\2.5.0\gems\logstash-output-s3-4.3.3\spec\outputs\s3_spec.rb
- added "file_prefix" => file_prefix at line 15
\vendor\bundle\jruby\2.5.0\gems\logstash-output-s3-4.3.3\spec\supports\helpers.rb
- added let(:file_prefix) {} at line line 12
- added "file_prefix" => file_prefix, at line 19
\vendor\bundle\jruby\2.5.0\gems\logstash-output-s3-4.3.3\lib\logstash\outputs\s3.rb
- added config :file_prefix, :validate => :string right after the statement config :prefix, :validate => :string, :default => '' at line 154
- added @file_prefix = file_prefix at line 226
Now when I start the logstash service I get this error message
Unknown setting 'file_prefix' for s3
I've searched and I've tried everything I can think but I can't get past this hurdle.
TIA,
Bill Youngman