Hello,
I am a newbie on ELK and recently I have installed LOGSTASH 1.5.3 on my ELASTICSEARCH 1.7 on one node on my WINDOWS LAPTOP. I need a plugin which is "logstash-filter-elasticsearch" ... found this on ruby.org, and after having modified gemfile installed it with plugin install --no-verify command.
Installation was successful therefore plugin list command does not give the plugin above. And logstah-conf execution failed wen calling filter plugin.
I have to say the plugin is logstash-filter-elasticsearch-0.1.6.
Here is my logstash conf :
input {
stdin { }
}
filter {
elasticsearch {
hosts => ["localhost"]
query => "_index:code2 AND _type:CODE2 AND CODE:A"
}
}
output {
elasticsearch {
action => "index"
host => "localhost"
port => "9200"
protocol => "http"
index => "code3"
workers => 1
}
stdout { codec => rubydebug }
Here is error msg :
"couldn't find any filter plugin named elasticsearch".
}
I am not able to see what is wrong. But I am not familiar with gem concept and local gem installation.
Thanks in advance for some clues.
That's what I did ... in 2 steps.
Fisrt I executed the plugin command as follows : plugin install " my folder\logstash-filter-elasticsearch-0.1.6.gem. It installed a local-gems folder below vendor : \logstash-1.5.3\vendor\local_gems\9cac55a4\logstash-filter-elasticsearch-0.1.6 with all gem/ ruby components and a gem file.
Second : I updated the gemfile with this :
gem "logstash-filter-elasticsearch" , :path => "my folder/logstash-1.5.3/vendor/local_gems/9cac55a4/logstash-filter-elasticsearch-0.1.6" and carried on with plugin install --no-verify comand with successg. Before I had to change \ by / as the command interpreted this as an escape character .
May be it works on linux environment and for windows 7 ?
Do you see anything wrong ... someone ?
I am confused.
Actually I am not able to install this plugin from ruby because of proxy access. So I have downloaded this gem on a folder on my laptop and then I have executed the plugin command install from logstash/bin specifying the folder where the gem is stored and so on ...When I downloaded it ... it is just the logstash-filter-elasticserach-0.1.6.gem which came to my laptop. I applied the install as I explained above and then the whole components were installed in folders.
So seems to have done the job successfully on installtion operation. But Logstash execution doesn't recognize this plugin in error message. May be the name of the plugin ? The folder where it is stored ? I really don't know because everything seems to be present in terms of components.
Could you please elaborate your reply a little more about how to install a gem plugin if you know the process ?
Thanks in advance.
logstash-1.5.2>bin\plugin list
(logstash-output-rdh is not listed)
What I don't get is the install says it succeeded, but no directory is getting created under logstash-1.5.2\vendor\bundle\jruby\1.9\gems\logstash-output-rdh-1.0.0
I am strugeling with this for quite som time. And guess i've found the problem now! \o/
I tried to build the gem for logstash-filter-geocoder and everything seems to work fine, it just was not available to logstash although it said "Installation successful". However in one time it worked and i could not remember what i did differently. But the working gem had a bigger filesize.. For the solution see here: Files in gemspec.
So creating a gem from a directory not controlled by git does create some kind of "empty" gem. That is way installation seems to be successful.
Now I succeed in installing logstash-elastics-seach-0.1.6 gem with this command line :
gem install logstash-filter-elasticsearch --http-proxy=http://user:pass@server:port.
It fetched a lot of thinks and at the end one gem installed successfully. I see my gem plugin installed in a gem shared folder. But no mix with other gem standard plugins. Even --no-verify command has no effect. And logstash execution failed on this gem plugin filter.
I tried a plugin install command line on this filter with several options but it failed with the same msg again and again ... want to download from rubygems.org ... therefore it has been already dowloaded in shared gem folders. So what can I do at this very point ? Please could you help with details ...I would appreciate. Thanks in advance.
Just to say I completely succeed in installing this gem plugin.
So here are the steps I have followed :
Download the gem from rubygems.org with this command line : gem install "your plugin" --http_proxy=http://user:pass@server:port
Copy & Paste 2 files from installation :
From /your folder/vendor\jruby\lib\ruby\gems\shared\gems
Copy logstash-filter-elasticsearch-0.1.6 folder to /your folder/logstash-1.5.3\vendor\bundle\jruby\1.9\gems folder (where all the other plugins are stored into)
expand logstash-filter-elasticsearch-0.1.6 and copy gemspec file from this folder to
/your folder/logstash-1.5.3\vendor\bundle\jruby\1.9\specifications
Modyfy s.file like this s.files = Dir.glob('vendor/*')
and then from /your folder/logstash-1.5.3\bin\plugin install your gem plugin.
Plugin list command shows you your plugin installed with "standards".
I think we can also download gem plugin directly and execute a local gem plugin install and then modify the s.file gemspec otherwise it tries to access git and it is a pain.
So it works for me now. Hope it helps.
Many thanks for people who gave me the clues in the posts above.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
