Logstash server read data from resturl


(Mayank Verma) #1

I want to read data from resturl and feed that in logstash server. i did google and found [a link] https://github.com/logstash-plugins/logstash-input-http_poller

Logstash version - logstash-1.4.5-1_a2bacae.noarch
plugin- https://github.com/logstash-plugins/logstash-input-http_poller

I tried it with my ELK setup but its is not working for me. while installing plugin (step 2.2) " bin/plugin install /home/elk/logstash-input-http_poller/logstash-input-http_poller-2.0.3.gem" it returned me "Can only install contrib at this time... Exiting."
When i start investigating my "plugin" script that i got from logstash rpm is very specific about $2 argument that can only be "contrib" if $2 is not "contrib" script will exit right away. So as per tutorial instruction section 2.2 passing your plugin gem file will always fail.

Can some one help me in this, i do'nt know what i'm missing to run it.


(Magnus B├Ąck) #2

You need Logstash 1.5 or later.


(Mayank Verma) #3

I upgraded logstash to 2.1, With this gem conflicts are comming

[elk@hostnamelogstash]$ sudo bin/plugin install /home/elk/logstash-input-http_poller/logstash-input-http_poller-2.0.3.gem
[sudo] password for elk:
Validating /home/elk/logstash-input-http_poller/logstash-input-http_poller-2.0.3.gem
Installing logstash-input-http_poller
Plugin version conflict, aborting
ERROR: Installation Aborted, message: Bundler could not find compatible versions for gem "logstash-mixin-http_client":
In snapshot (Gemfile.lock):
logstash-mixin-http_client (= 2.0.3)

In Gemfile:
logstash-mixin-http_client (< 3.0.0, >= 2.0.2) java

logstash-input-http_poller (= 2.0.3) java depends on
  logstash-mixin-http_client (< 3.0.0, >= 2.1.0) java

Running bundle update will rebuild your snapshot from scratch, using only
the gems in your Gemfile, which may resolve the conflict.

I tried doing bundle upadte for "logstash-input-http_poller" plugin after update it gives me.

[elk@hostname logstash-input-http_poller]$ bundle update
Fetching gem metadata from https://rubygems.org/..........
Fetching version metadata from https://rubygems.org/...
Fetching dependency metadata from https://rubygems.org/..
Resolving dependencies...........
Installing rake 10.5.0 (was 10.4.2)
Using cabin 0.7.2
Using clamp 0.6.5
Using coderay 1.1.0
Using concurrent-ruby 0.9.2
Using diff-lcs 1.2.5
Using ffi 1.9.10
Using filesize 0.0.4
Using flores 0.0.6
Using gem_publisher 1.5.0
Using gems 0.8.3
Using i18n 0.6.9
Using insist 1.0.0
Using jar-dependencies 0.3.1
Using jrjackson 0.3.8
Using jruby-openssl 0.9.13
Using kramdown 1.9.0
Using logstash-core-event 2.2.0.snapshot2
Using minitar 0.5.4
Using method_source 0.8.2
Using slop 3.6.0
Using rubyzip 1.1.7
Using stud 0.0.22
Using thread_safe 0.3.5
Using polyglot 0.3.5
Using rspec-support 3.1.2
Using bundler 1.11.2
Using spoon 0.0.4
Using manticore 0.5.2
Using treetop 1.4.15
Using rspec-core 3.1.7
Using rspec-expectations 3.1.2
Using rspec-mocks 3.1.3
Using pry 0.10.3
Using rspec 3.1.0
Using logstash-core 2.2.0.snapshot2
Using rspec-wait 0.0.8
Using logstash-codec-json 2.0.4
Using logstash-codec-plain 2.0.2
Using logstash-devutils 0.0.18
Using logstash-mixin-http_client 2.2.0
Using logstash-input-http_poller 2.0.3 from source at .
Bundle updated!

later i tried bundle update with logstash it returned me

[elk@hostname logstash]$ bundle update
Ignoring executable-hooks-1.3.2 because its extensions are not built. Try: gem pristine executable-hooks --version 1.3.2
Ignoring gem-wrappers-1.2.7 because its extensions are not built. Try: gem pristine gem-wrappers --version 1.2.7
Ignoring jruby-launcher-1.1.1-java because its extensions are not built. Try: gem pristine jruby-launcher --version 1.1.1
Ignoring executable-hooks-1.3.2 because its extensions are not built. Try: gem pristine executable-hooks --version 1.3.2
Ignoring gem-wrappers-1.2.7 because its extensions are not built. Try: gem pristine gem-wrappers --version 1.2.7
Ignoring jruby-launcher-1.1.1-java because its extensions are not built. Try: gem pristine jruby-launcher --version 1.1.1
Fetching gem metadata from https://rubygems.org/........
Fetching version metadata from https://rubygems.org/...
Fetching dependency metadata from https://rubygems.org/..
Resolving dependencies....................................................
Bundler could not find compatible versions for gem "aws-sdk-resources":
In Gemfile:
logstash-input-sqs java was resolved to 2.0.1, which depends on
logstash-mixin-aws (>= 1.0.0) java was resolved to 2.0.1, which depends on
aws-sdk (~> 2.1.0) java was resolved to 2.1.0, which depends on
aws-sdk-resources (= 2.1.0) java

logstash-output-cloudwatch java was resolved to 2.0.2, which depends on
  aws-sdk java was resolved to 2.1.0, which depends on
    aws-sdk-resources (= 2.2.10) java

Bundler could not find compatible versions for gem "logstash":
In Gemfile:
logstash-input-sqs java was resolved to 0.1.0, which depends on
logstash (< 2.0.0, >= 1.4.0) java

Could not find gem 'logstash (< 2.0.0, >= 1.4.0) java', which is required by gem
'logstash-input-sqs java', in any of the sources.Bundler could not find
compatible versions for gem "logstash-core":
In Gemfile:
logstash-core (= 2.1.1) java

logstash-input-beats java was resolved to 2.1.2, which depends on

logstash-codec-multiline (~> 2.0.5) java was resolved to 2.0.6, which
depends on
logstash-core (< 3.0.0, >= 2.0.0) java

   logstash-input-pipe java was resolved to 2.0.2, which depends on
  logstash-core (< 3.0.0, >= 2.0.0.beta2) java

logstash-input-sqs java was resolved to 0.1.3, which depends on
  logstash-core (< 2.0.0, >= 1.4.0) java

What next i can do to resolve this?


(Mark Walkom) #4

Do you need to do an offline install?


(Mayank Verma) #5

whats your mean with offline install? I have logstash server running on one of vm i have and same vm i am installing http poller plugin to read rest urls input. may this will help you to understand what i am trying.


(Mark Walkom) #6

Is an offline install.

Just use bin/plugin install logstash-input-http_poller.


(Mayank Verma) #7

Installation is successful now. but now i am not able to run logstash server. in logs it says : {:timestamp=>"2016-01-19T14:59:13.265000+0530", :message=>"The error reported is: \n Address already in use - bind - Address already in use"}

I can see installed plugin in bin/plugin list , it is listing
[elk@hostnamelogstash]$ bin/plugin list --verbose
logstash-codec-collectd (2.0.2)
logstash-codec-dots (2.0.2)
logstash-codec-edn (2.0.2)
logstash-codec-edn_lines (2.0.2)
logstash-codec-es_bulk (2.0.2)
logstash-codec-fluent (2.0.2)
logstash-codec-graphite (2.0.2)
logstash-codec-json (2.0.4)
logstash-codec-json_lines (2.0.2)
logstash-codec-line (2.0.2)
logstash-codec-msgpack (2.0.2)
logstash-codec-multiline (2.0.4)
logstash-codec-netflow (2.0.2)
logstash-codec-oldlogstashjson (2.0.2)
logstash-codec-plain (2.0.2)
logstash-codec-rubydebug (2.0.4)
logstash-filter-anonymize (2.0.2)
logstash-filter-checksum (2.0.2)
logstash-filter-clone (2.0.4)
logstash-filter-csv (2.1.0)
logstash-filter-date (2.0.2)
logstash-filter-dns (2.0.2)
logstash-filter-drop (2.0.2)
logstash-filter-fingerprint (2.0.2)
logstash-filter-geoip (2.0.4)
logstash-filter-grok (2.0.2)
logstash-filter-json (2.0.2)
logstash-filter-kv (2.0.2)
logstash-filter-metrics (3.0.0)
logstash-filter-multiline (2.0.3)
logstash-filter-mutate (2.0.2)
logstash-filter-ruby (2.0.2)
logstash-filter-sleep (2.0.2)
logstash-filter-split (2.0.2)
logstash-filter-syslog_pri (2.0.2)
logstash-filter-throttle (2.0.2)
logstash-filter-urldecode (2.0.2)
logstash-filter-useragent (2.0.3)
logstash-filter-uuid (2.0.3)
logstash-filter-xml (2.0.2)
logstash-input-beats (2.0.3)
logstash-input-couchdb_changes (2.0.2)
logstash-input-elasticsearch (2.0.2)
logstash-input-eventlog (3.0.1)
logstash-input-exec (2.0.4)
logstash-input-file (2.0.3)
logstash-input-ganglia (2.0.4)
logstash-input-gelf (2.0.2)
logstash-input-generator (2.0.2)
logstash-input-graphite (2.0.4)
logstash-input-heartbeat (2.0.2)
logstash-input-http (2.0.2)
logstash-input-http_poller (2.0.2)
logstash-input-imap (2.0.2)
logstash-input-irc (2.0.3)
logstash-input-jdbc (2.0.5)
logstash-input-kafka (2.0.2)
logstash-input-log4j (2.0.4)
logstash-input-lumberjack (2.0.5)
logstash-input-pipe (2.0.2)
logstash-input-rabbitmq (3.1.1)
logstash-input-redis (2.0.2)
logstash-input-s3 (2.0.3)
logstash-input-snmptrap (2.0.2)
logstash-input-sqs (2.0.3)
logstash-input-stdin (2.0.2)
logstash-input-syslog (2.0.2)
logstash-input-tcp (3.0.0)
logstash-input-twitter (2.2.0)
logstash-input-udp (2.0.3)
logstash-input-unix (2.0.4)
logstash-input-xmpp (2.0.3)
logstash-input-zeromq (2.0.2)
logstash-output-cloudwatch (2.0.2)
logstash-output-csv (2.0.2)
logstash-output-elasticsearch (2.2.0)
logstash-output-email (3.0.2)
logstash-output-exec (2.0.2)
logstash-output-file (2.2.0)
logstash-output-ganglia (2.0.2)
logstash-output-gelf (2.0.2)
logstash-output-graphite (2.0.2)
logstash-output-hipchat (3.0.2)
logstash-output-http (2.0.5)
logstash-output-irc (2.0.2)
logstash-output-juggernaut (2.0.2)
logstash-output-kafka (2.0.1)
logstash-output-lumberjack (2.0.4)
logstash-output-nagios (2.0.2)
logstash-output-nagios_nsca (2.0.3)
logstash-output-null (2.0.2)
logstash-output-opentsdb (2.0.2)
logstash-output-pagerduty (2.0.2)
logstash-output-pipe (2.0.2)
logstash-output-rabbitmq (3.0.6)
logstash-output-redis (2.0.2)
logstash-output-s3 (2.0.3)
logstash-output-sns (3.0.2)
logstash-output-sqs (2.0.2)
logstash-output-statsd (2.0.4)
logstash-output-stdout (2.0.3)
logstash-output-tcp (2.0.2)
logstash-output-udp (2.0.2)
logstash-output-xmpp (2.0.2)
logstash-output-zeromq (2.0.2)
logstash-patterns-core (2.0.2)


(Mark Walkom) #8

Would suggest something else is using whatever port you have specified in the LS conf.


(system) #9