Twitter river(s) stopped working on October 6


(pedro) #1

We recently noticed that our twitter rivers stopped working on oct
6th. We're using it at mozilla to do sentiment analysis, and we have 2
servers that stopped working at the same time.

(and by at the same time, I literally mean the last commited tweet was
the same: created_at: 2011-10-06T17:35:25.000Z )

I've seen 2 types of erros:

[2011-10-12 14:21:29,106][WARN ][river.twitter ] [Colleen
Wing] [twitter][twitter_river] stream failure, restarting stream...
Connection refusedTwitterException{exceptionCode=[39caf88f-15b6fe9c
39caf88f-15b6fe72], statusCode=-1, retryAfter=0, rateLimitStatus=null,
version=2.1.10}
at
twitter4j.internal.http.HttpClientImpl.request(HttpClientImpl.java:
325)
at
twitter4j.internal.http.HttpClientWrapper.request(HttpClientWrapper.java:
72)
at
twitter4j.internal.http.HttpClientWrapper.post(HttpClientWrapper.java:
103)
at twitter4j.TwitterStream.getFilterStream(TwitterStream.java:459)
at twitter4j.TwitterStream$8.getStream(TwitterStream.java:440)
at twitter4j.TwitterStream$8.getStream(TwitterStream.java:438)
at twitter4j.TwitterStream
$TwitterStreamConsumer.run(TwitterStream.java:673)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:
195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at sun.net.NetworkClient.doConnect(NetworkClient.java:158)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:394)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)
at sun.net.www.http.HttpClient.(HttpClient.java:233)
at sun.net.www.http.HttpClient.New(HttpClient.java:306)
at sun.net.www.http.HttpClient.New(HttpClient.java:323)
at
sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:
975)
at
sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:
916)
at
sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:
841)
at
sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:
1019)
at
twitter4j.internal.http.HttpClientImpl.request(HttpClientImpl.java:
283)
... 6 more
^C

and

[2011-10-12 23:07:05,273][WARN ][http.netty ] [Gyrich,
Henry Peter] Caught exception while handling client http traffic,
closing connection
java.io.IOException: Connection timed out
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:202)
at sun.nio.ch.IOUtil.read(IOUtil.java:169)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:
243)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:
321)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:
280)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
200)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

Any tips before I turn on debug mode? I'm doing some tests with the
latest snapshot version of twitter4j, but if anyone has tips on this
would be helpful

-pedro


(Bowen Masco) #2

I am not using the twitter river plugin, but I did had a similar
problem with a different twitter app.

It was caused by twitter's switch to ssl only( http to https urls)

https://dev.twitter.com/blog/streaming-api-turning-ssl-only-september-29th

The fix just required changing all the http urls to https urls for twitter.

Bowen


(pedro) #3

I'll look into the twitter river code to see what's currently using, thanks
for the tip

On Wednesday, October 12, 2011, Bowen Masco bowen@codingfoo.com wrote:

I am not using the twitter river plugin, but I did had a similar
problem with a different twitter app.

It was caused by twitter's switch to ssl only( http to https urls)

https://dev.twitter.com/blog/streaming-api-turning-ssl-only-september-29th

The fix just required changing all the http urls to https urls for
twitter.

Bowen


(Karussell) #4

The twitter4j needs an update as I pointed out (on twitter).

created an issue here

On 13 Okt., 00:00, Pedro Alves pmgal...@gmail.com wrote:

I'll look into the twitter river code to see what's currently using, thanks
for the tip

On Wednesday, October 12, 2011, Bowen Masco bo...@codingfoo.com wrote:

I am not using the twitter river plugin, but I did had a similar
problem with a different twitter app.

It was caused by twitter's switch to ssl only( http to https urls)

https://dev.twitter.com/blog/streaming-api-turning-ssl-only-september...

The fix just required changing all the http urls to https urls for
twitter.

Bowen


(pedro) #5

Perfect!

I'll try again, just replacing the jar didn't work (even though it was
2am, don't even remember what I did). Also didn't find any classpath
references to the libs, but can also try to recompile from source to
see if it works

Thanks, I'll let know how it goes

On Thu Oct 13 08:32:29 2011, Karussell wrote:

The twitter4j needs an update as I pointed out (on twitter).

created an issue here

https://github.com/elasticsearch/elasticsearch/issues/1393

On 13 Okt., 00:00, Pedro Alvespmgal...@gmail.com wrote:

I'll look into the twitter river code to see what's currently using, thanks
for the tip

On Wednesday, October 12, 2011, Bowen Mascobo...@codingfoo.com wrote:

I am not using the twitter river plugin, but I did had a similar
problem with a different twitter app.

It was caused by twitter's switch to ssl only( http to https urls)

https://dev.twitter.com/blog/streaming-api-turning-ssl-only-september...

The fix just required changing all the http urls to https urls for
twitter.

Bowen


(pedro) #6

Got it. - thanks.

Created pull request to fix the issue

On Thu Oct 13 10:41:51 2011, Pedro Alves wrote:

Perfect!

I'll try again, just replacing the jar didn't work (even though it was
2am, don't even remember what I did). Also didn't find any classpath
references to the libs, but can also try to recompile from source to
see if it works

Thanks, I'll let know how it goes

On Thu Oct 13 08:32:29 2011, Karussell wrote:

The twitter4j needs an update as I pointed out (on twitter).

created an issue here

https://github.com/elasticsearch/elasticsearch/issues/1393

On 13 Okt., 00:00, Pedro Alvespmgal...@gmail.com wrote:

I'll look into the twitter river code to see what's currently using,
thanks
for the tip

On Wednesday, October 12, 2011, Bowen Mascobo...@codingfoo.com wrote:

I am not using the twitter river plugin, but I did had a similar
problem with a different twitter app.

It was caused by twitter's switch to ssl only( http to https urls)

https://dev.twitter.com/blog/streaming-api-turning-ssl-only-september...

The fix just required changing all the http urls to https urls for
twitter.

Bowen


(system) #7