Issue Developing Filter Plugin on Windows

Elastic Stack Logstash
1

I am trying to write a new logstash filter, and I can't seem to get the rspec tests to run. I followed the guide posted here: https://www.elastic.co/guide/en/logstash/current/filter-new-plugin.html

this is my process:

PS C:\LS\logstash-7.5.2> jruby -v
jruby 9.2.11.1 (2.5.7) 2020-03-25 b1f55b1a40 Java HotSpot(TM) 64-Bit Server VM 25.201-b09 on 1.8.0_201-b09 +jit [mswin32-x86_64]
PS C:\LS\logstash-7.5.2> .\bin\logstash-plugin.bat generate --type filter --name example                                 Creating C:/LS/logstash-7.5.2/logstash-filter-example
         create logstash-filter-example/CHANGELOG.md
         create logstash-filter-example/CONTRIBUTORS
         create logstash-filter-example/DEVELOPER.md
         create logstash-filter-example/docs/index.asciidoc
         create logstash-filter-example/Gemfile
         create logstash-filter-example/lib/logstash/filters/example.rb
         create logstash-filter-example/LICENSE
         create logstash-filter-example/logstash-filter-example.gemspec
         create logstash-filter-example/Rakefile
         create logstash-filter-example/README.md
         create logstash-filter-example/spec/filters/example_spec.rb
         create logstash-filter-example/spec/spec_helper.rb
PS C:\LS\logstash-7.5.2> Add-Content -Path .\logstash-filter-example\Gemfile -Value 'gem "logstash-core", :github => "elastic/logstash", :branch => "7.5"'
PS C:\LS\logstash-7.5.2> gc .\logstash-filter-example\Gemfile
source 'https://rubygems.org'
gemspec

gem "logstash-core", :github => "elastic/logstash", :branch => "7.5"

The guide says this should be gem "logstash", but that returns an error about the "logstash" gem not existing, "logstash-core" seems to be the correct line (maybe the docs are out of date?)
I then add a summary/description/homepage to the .gemspec file

PS C:\LS\logstash-7.5.2\logstash-filter-example> bundle install
Fetching https://github.com/elastic/logstash.git
Updating files: 100% (2818/2818), done.
Fetching gem metadata from https://rubygems.org/.........
Resolving dependencies...................................
Using rake 13.0.1
Using bundler 2.1.4
Using numerizer 0.1.1
Using chronic_duration 0.10.6
Using clamp 0.6.5
Using coderay 1.1.2
Using concurrent-ruby 1.1.6
Using diff-lcs 1.3
Using multi_json 1.14.1
Using elasticsearch-api 5.0.5
Using multipart-post 2.1.1
Using faraday 1.0.1
Using elasticsearch-transport 5.0.5
Using elasticsearch 5.0.5
Using ffi 1.12.2 (java)
Using filesize 0.2.0
Using fivemat 1.3.7
Using gem_publisher 1.5.0
Using gems 1.2.0
Using i18n 1.8.2
Fetching jrjackson 0.4.11 (java)
Installing jrjackson 0.4.11 (java)
Using jruby-openssl 0.10.4 (java)
Using kramdown 1.14.0
Using openssl_pkcs8_pure 0.0.0.2
Using manticore 0.6.4 (java)
Using minitar 0.9
Using method_source 1.0.0
Using spoon 0.0.6
Using pry 0.13.1 (java)
Fetching puma 2.16.0 (java)
Installing puma 2.16.0 (java)
Fetching rack 1.6.13
Installing rack 1.6.13
Using rubyzip 1.3.0
Fetching rack-protection 1.5.5
Installing rack-protection 1.5.5
Using tilt 2.0.10
Fetching sinatra 1.4.8
Installing sinatra 1.4.8
Using stud 0.0.23
Using thread_safe 0.3.6 (java)
Using polyglot 0.3.5
Using treetop 1.6.10
Using logstash-core 7.5.3 (java) from https://github.com/elastic/logstash.git (at 7.5@32e3ff2)
Using logstash-core-plugin-api 2.1.16 (java) from https://github.com/elastic/logstash.git (at 7.5@32e3ff2)
Using logstash-codec-plain 3.0.6
Using rspec-support 3.9.3
Using rspec-core 3.9.2
Using rspec-expectations 3.9.2
Using rspec-mocks 3.9.1
Using rspec 3.9.0
Using rspec-wait 0.0.9
Using logstash-devutils 2.0.3 (java)
Using logstash-filter-example 0.1.0 from source at `.`
Bundle complete! 3 Gemfile dependencies, 50 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.

This is the point where things fail. If I try the command the guide says, it claims that rspec isn't installed, when it clearly was just installed in the previous command:

PS C:\LS\logstash-7.5.2\logstash-filter-example> bundle exec rspec
bundler: command not found: rspec
Install missing gem executables with `bundle install`

I did a bit of searching and it seems like I have to pass the directory to rspec, but when I try this command I get an error:

PS C:\LS\logstash-7.5.2\logstash-filter-example> bundle exec rspec spec

An error occurred while loading ./spec/filters/example_spec.rb.
e[31mFailure/Error: require "logstash/devutils/rspec/spec_helper"e[0m
e[31me[0m
e[31mNameError:e[0m
e[31m  missing class name (`org.logstash.RubyUtil')e[0m
e[36m# ./spec/spec_helper.rb:2:in `<main>'e[0m
e[36m# ./spec/filters/example_spec.rb:2:in `<main>'e[0m
e[36m# C:\tools\jruby-9.2\bin\rspec:23:in `<main>'e[0m
No examples found.


Finished in 0.001 seconds (files took 0.737 seconds to load)
e[31m0 examples, 0 failures, 1 error occurred outside of examplese[0m

I haven't changed anything from the generated spec files. Can someone help? I found a similar post from 2018 without any replies: Creating an output plugin -- guide not working

Thanks

2

Plugin development works out better if you are working with logstash source instead of the gems from rubygems. After some help from @yaauie, this solved my issue with running plugin tests from the windows filesystem:

mkdir logstash-dev # created a root development directory
cd .\logstash-dev
git clone https://github.com/elastic/logstash.git --depth 1 -b 7.x # shallow cloned the 7.x logstash branch
cd .\logstash
.\gradlew.bat assemble # run gradle to get Logstash to an executable state

$env:LOGSTASH_SOURCE = '1' # true
$env:LOGSTASH_PATH = 'D:\logstash-dev\logstash' # full path to logstash source

cd ..
mkdir logstash-plugins
cd .\logstash-plugins

To quickly get the directory structure for a logstash plugin, clone from the example filter on github

git clone https://github.com/logstash-plugins/logstash-filter-example

The important part is that the GemFile tells bundler to use logstash from source when the environment variables are set:

source 'https://rubygems.org'

gemspec

logstash_path = ENV["LOGSTASH_PATH"] || "../../logstash"
use_logstash_source = ENV["LOGSTASH_SOURCE"] && ENV["LOGSTASH_SOURCE"].to_s == "1"

if Dir.exist?(logstash_path) && use_logstash_source
  gem 'logstash-core', :path => "#{logstash_path}/logstash-core"
  gem 'logstash-core-plugin-api', :path => "#{logstash_path}/logstash-core-plugin-api"
end

cd .\logstash-filter-example
bundle install
bundle exec rake vendor
bundle exec rspec spec --no-color # helps with windows console output formatting

The tests ran successfully!

Sending Logstash logs to null which is now configured via log4j2.properties
Run options: exclude {:integration=>true, :redis=>true, :socket=>true, :performance=>true, :couchdb=>true, :elasticsearch=>true, :elasticsearch_secure=>true, :export_cypher=>true, :unix=>true}

Randomized with seed 700
.

Finished in 4.96 seconds (files took 36.81 seconds to load)
1 example, 0 failures

Randomized with seed 700
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.