Logstash 1.5.3 gem plugins

(RENEE) #1

I am a newbie on ELK and recently I have installed LOGSTASH 1.5.3 on my ELASTICSEARCH 1.7 on one node on my WINDOWS LAPTOP. I need a plugin which is "logstash-filter-elasticsearch" ... found this on ruby.org, and after having modified gemfile installed it with plugin install --no-verify command.
Installation was successful therefore plugin list command does not give the plugin above. And logstah-conf execution failed wen calling filter plugin.
I have to say the plugin is logstash-filter-elasticsearch-0.1.6.
Here is my logstash conf :
input {
stdin { }

filter {
elasticsearch {
hosts => ["localhost"]
query => "_index:code2 AND _type:CODE2 AND CODE:A"

output {
elasticsearch {
action => "index"
host => "localhost"
port => "9200"
protocol => "http"
index => "code3"
workers => 1
stdout { codec => rubydebug }
Here is error msg :
"couldn't find any filter plugin named elasticsearch".


I am not able to see what is wrong. But I am not familiar with gem concept and local gem installation.
Thanks in advance for some clues.

(Mark Walkom) #2

Any reason you didn't use the plugin command to install this?

(RENEE) #3

That's what I did ... in 2 steps.
Fisrt I executed the plugin command as follows : plugin install " my folder\logstash-filter-elasticsearch-0.1.6.gem. It installed a local-gems folder below vendor : \logstash-1.5.3\vendor\local_gems\9cac55a4\logstash-filter-elasticsearch-0.1.6 with all gem/ ruby components and a gem file.
Second : I updated the gemfile with this :
gem "logstash-filter-elasticsearch" , :path => "my folder/logstash-1.5.3/vendor/local_gems/9cac55a4/logstash-filter-elasticsearch-0.1.6" and carried on with plugin install --no-verify comand with successg. Before I had to change \ by / as the command interpreted this as an escape character .
May be it works on linux environment and for windows 7 ?
Do you see anything wrong ... someone ?

(Mark Walkom) #4

I mean plugin instal logstash-filter-elasticsearch and then download it?

(RENEE) #5

Hello Mark,

I am confused.
Actually I am not able to install this plugin from ruby because of proxy access. So I have downloaded this gem on a folder on my laptop and then I have executed the plugin command install from logstash/bin specifying the folder where the gem is stored and so on ...When I downloaded it ... it is just the logstash-filter-elasticserach-0.1.6.gem which came to my laptop. I applied the install as I explained above and then the whole components were installed in folders.
So seems to have done the job successfully on installtion operation. But Logstash execution doesn't recognize this plugin in error message. May be the name of the plugin ? The folder where it is stored ? I really don't know because everything seems to be present in terms of components.
Could you please elaborate your reply a little more about how to install a gem plugin if you know the process ?
Thanks in advance.

(Ed Hammond) #6

I am trying to do something similar & having no joy.

Building a local plugin (copied logstash-output-http as an example) and then trying to install on the local Logstash instance on a Win7 machine.

Did all the commands I think as documented in various places ...

jruby -S gem install bundler

gem build logstash-output-rdh.gemspec
Successfully built RubyGem
Name: logstash-output-rdh
Version: 1.0.0
File: logstash-output-rdh-1.0.0.gem

plugin install logstash-output-rdh-1.0.0.gem
Validating logstash-output-rdh-1.0.0.gem
Installing logstash-output-rdh
Installation successful

logstash-1.5.2>bin\plugin list
(logstash-output-rdh is not listed)

What I don't get is the install says it succeeded, but no directory is getting created under logstash-1.5.2\vendor\bundle\jruby\1.9\gems\logstash-output-rdh-1.0.0

Where might I look for more guidance?

(Tobias Bradtke) #7

I am strugeling with this for quite som time. And guess i've found the problem now! \o/
I tried to build the gem for logstash-filter-geocoder and everything seems to work fine, it just was not available to logstash although it said "Installation successful". However in one time it worked and i could not remember what i did differently. But the working gem had a bigger filesize.. For the solution see here: Files in gemspec.

So creating a gem from a directory not controlled by git does create some kind of "empty" gem. That is way installation seems to be successful.

(Eric Deveaud) #8

you don't have to be under git control to create a gem.

you just must have the s.files in gemspec file correctly set.
this can be done in various way even manually :wink:

(RENEE) #9


Now I succeed in installing logstash-elastics-seach-0.1.6 gem with this command line :
gem install logstash-filter-elasticsearch --http-proxy=http://user:pass@server:port.
It fetched a lot of thinks and at the end one gem installed successfully. I see my gem plugin installed in a gem shared folder. But no mix with other gem standard plugins. Even --no-verify command has no effect. And logstash execution failed on this gem plugin filter.
I tried a plugin install command line on this filter with several options but it failed with the same msg again and again ... want to download from rubygems.org ... therefore it has been already dowloaded in shared gem folders. So what can I do at this very point ? Please could you help with details ...I would appreciate. Thanks in advance.

(RENEE) #10


Just to say I completely succeed in installing this gem plugin.
So here are the steps I have followed :

  1. Download the gem from rubygems.org with this command line : gem install "your plugin" --http_proxy=http://user:pass@server:port
  2. Copy & Paste 2 files from installation :
  • From /your folder/vendor\jruby\lib\ruby\gems\shared\gems
    Copy logstash-filter-elasticsearch-0.1.6 folder to /your folder/logstash-1.5.3\vendor\bundle\jruby\1.9\gems folder (where all the other plugins are stored into)
  • expand logstash-filter-elasticsearch-0.1.6 and copy gemspec file from this folder to
    /your folder/logstash-1.5.3\vendor\bundle\jruby\1.9\specifications
  1. Modyfy s.file like this s.files = Dir.glob('vendor/*')
  2. and then from /your folder/logstash-1.5.3\bin\plugin install your gem plugin.
  3. Plugin list command shows you your plugin installed with "standards".

I think we can also download gem plugin directly and execute a local gem plugin install and then modify the s.file gemspec otherwise it tries to access git and it is a pain.

So it works for me now. Hope it helps.
Many thanks for people who gave me the clues in the posts above.

(system) #11