Hitting Rate limit on Twitter River when creating multiple rivers


(Amit Gupta) #1

Hi All,

I have a 3-node elastic search cluster which is running fine with twitter
river. I want to three different rivers :-

  1. the public stream
  2. a filtered stream based on keywords
  3. a filtered stream based on selected users

However, whenever I start more than one river, I start receiving multiple
errors related to rate limit :-

Returned by the Streaming API:

  • Too many login attempts in a short period of time.*

  • Running too many copies of the same application authenticating with the
    same account name.*

Easy there, Turbo. Too many requests recently. Enhance your calm.

[2013-12-11 23:13:33,701][WARN ][river.twitter ] [Hyde]
[twitter][my_twitter_river2] stream failure, restarting stream...

420:Returned by the Search and Trends API when you are being rate limited
(https://dev.twitter.com/docs/rate-limiting).

I was wondering if its normal behavior (I do know that streaming api can
return something similar if there are too many tweets matching the
criterion) but I am not sure if I should be receiving them so frequently.
Also, is it ok to have multiple rivers using the same oauth, or is it
better to create different users/app for each river.

thanks,

Amit

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ea32550c-054d-46a7-bde8-004dfc41bc8a%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(David Pilato) #2

From Twitter API point of view, this is a normal behavior.
When I need to do that, I create multiples Twitter Account (and applications).

That said I think you have to pay Twitter for the firehose.
For free, you can have sample stream and a reasonable amount of filtered tweets.

HTH

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 11 déc. 2013 à 23:15, Amit Gupta amitgupta151@gmail.com a écrit :

Hi All,

I have a 3-node elastic search cluster which is running fine with twitter river. I want to three different rivers :-

  1. the public stream
  2. a filtered stream based on keywords
  3. a filtered stream based on selected users

However, whenever I start more than one river, I start receiving multiple errors related to rate limit :-

Returned by the Streaming API:

Too many login attempts in a short period of time.

Running too many copies of the same application authenticating with the same account name.

Easy there, Turbo. Too many requests recently. Enhance your calm.

[2013-12-11 23:13:33,701][WARN ][river.twitter ] [Hyde] [twitter][my_twitter_river2] stream failure, restarting stream...

420:Returned by the Search and Trends API when you are being rate limited (https://dev.twitter.com/docs/rate-limiting).

I was wondering if its normal behavior (I do know that streaming api can return something similar if there are too many tweets matching the criterion) but I am not sure if I should be receiving them so frequently. Also, is it ok to have multiple rivers using the same oauth, or is it better to create different users/app for each river.

thanks,

Amit

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ea32550c-054d-46a7-bde8-004dfc41bc8a%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/88C3A05E-0A84-4EB1-BDD0-1EDC42666EE1%40pilato.fr.
For more options, visit https://groups.google.com/groups/opt_out.


(system) #3