Logstash twitter 2.2.0 input jruby error NoMethodError

Elastic Stack Logstash
#1

I was able to get my ELK setup to import the samples using the twitter input plugin but when I changed the conf to use my tags, follows I was able to pass the --configtest but logstash.log filled up with these errors

Does it mean my twitter API requests are unauthorized and if so do I need to have them multiple times for each input?

I'm running
elastic "number" : "2.1.0",
with the latest logstash plugins (I ran /opt/logstash/bin/plugin --update before I started with the sample tweets)
Ubuntu 14

My Logstash config
input { twitter { # add your data consumer_key => "" consumer_secret => "" oauth_token => "" oauth_token_secret => "" tags => ["MUNvLEI","MunVLei"] follows => ["ChampionsCup","skysportsrugby","btsportrugby","rugbytonight"] full_tweet => true type => "Media" } twitter { # add your data consumer_key => " consumer_secret => "" oauth_token => "" oauth_token_secret => "" tags => ["SUAF","16thman"] follows => ["munsterrugby","MRSC16"] full_tweet => true type => "Munster" } twitter { # add your data consumer_key => " consumer_secret => "" oauth_token => "" oauth_token_secret => "" tags => ["Tigersfamily"] follows => ["leicestertigers"] full_tweet => true type => "Leicester" } } output { elasticsearch { hosts => "localhost" index => "twitter_gis" document_type => "tweet_ls" } }

Output from Logstash.log
{:timestamp=>"2015-12-09T23:39:33.893000+0000", :message=>"", :exception=>Twitter::Error::Unauthorized, :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/twitter-5.15.0/lib/twitter/streaming/response.rb:21:inon_headers_complete'", "org/ruby_http_parser/RubyHttpParser.java:370:in <<'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/twitter-5.15.0/lib/twitter/streaming/response.rb:16:in<<'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-twitter-2.2.0/lib/logstash/inputs/twitter/patches.rb:31:in stream'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-twitter-2.2.0/lib/logstash/inputs/twitter/patches.rb:58:inrequest'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/twitter-5.15.0/lib/twitter/streaming/client.rb:37:in filter'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-twitter-2.2.0/lib/logstash/inputs/twitter.rb:128:inrun'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.0-java/lib/logstash/pipeline.rb:205:in inputworker'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.0-java/lib/logstash/pipeline.rb:198:instart_input'"], :options=>nil, :level=>:warn}`

#2

That means your auth details are wrong somewhere. You need to put them against each input section.

#3

Is it possible to include the auth details once to avoid copy & paste errors or make the input section cleaner ?

#4

I have the same API values in each of the 3 inputs
Do I need a different twitter api set for each twitter{} entry in the input part of the config?

#5

You shouldn't, but I don't know for certain.

#6

I don't think its a problem with my twitter credentials as when I have use_samples => true there are no errors in the log file and my index is populated with tweets but when I remove it, and delete the index I get a rapidly filling log file and my index isn't created

#7

Hi all,

I'm facing the same issue and I know for sure it's not coming from my credentials as I'm using them in two other configuration files and it works perfectly.

The only differences with the last one is that I'm using follows instead of keywords as I want to retrieve the twitter feed from an account and not only from tags.

Here is the input part of my configuration file:
input { twitter { consumer_key => "xxx" consumer_secret => "xxx" oauth_token => "xxx" oauth_token_secret => "xxx" follows => [ "@elastic" ] full_tweet => true } }

And here is my stacktrace:

{:exception=>Twitter::Error::Unauthorized, :backtrace=>["/Users/audrey/Dev/logstash-2.1.1/vendor/bundle/jruby/1.9/gems/twitter-5.15.0/lib/twitter/streaming/response.rb:21:inon_headers_complete'", "org/ruby_http_parser/RubyHttpParser.java:370:in <<'", "/Users/audrey/Dev/logstash-2.1.1/vendor/bundle/jruby/1.9/gems/twitter-5.15.0/lib/twitter/streaming/response.rb:16:in<<'", "/Users/audrey/Dev/logstash-2.1.1/vendor/bundle/jruby/1.9/gems/logstash-input-twitter-2.2.0/lib/logstash/inputs/twitter/patches.rb:31:in stream'", "/Users/audrey/Dev/logstash-2.1.1/vendor/bundle/jruby/1.9/gems/logstash-input-twitter-2.2.0/lib/logstash/inputs/twitter/patches.rb:58:inrequest'", "/Users/audrey/Dev/logstash-2.1.1/vendor/bundle/jruby/1.9/gems/twitter-5.15.0/lib/twitter/streaming/client.rb:37:in filter'", "/Users/audrey/Dev/logstash-2.1.1/vendor/bundle/jruby/1.9/gems/logstash-input-twitter-2.2.0/lib/logstash/inputs/twitter.rb:128:inrun'", "/Users/audrey/Dev/logstash-2.1.1/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.1-java/lib/logstash/pipeline.rb:206:in inputworker'", "/Users/audrey/Dev/logstash-2.1.1/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.1-java/lib/logstash/pipeline.rb:199:instart_input'"], :options=>nil, :level=>:warn}`

Perhaps I'm not using the follows attribute the right way? I've tried with or without the @ before the id but it makes no difference.

I've also tried to add use_samples => true as mentionned by @mob and it works but indexed tweets are not relevant.

Any help highly appreciated :slight_smile:

#8

I'm experiencing this issue as well. My time is correct and I've tested and verified my tokens and keys with Twitter's OAuth tool.

I'm using logstash 2.1.1 and logstash-input-twitter-2.2.0

Mike

#9

when you are using "follows" parameter you need to include twitter user's ID instead of user name or screen name.

you can use this site to grab relevant twitter id to username or twitter API.

follows => ["372841707"]
will do the fix for you.

3 Likes
#10