Twitter river(s) stopped working on October 6

pedro · October 12, 2011, 9:36pm

We recently noticed that our twitter rivers stopped working on oct
6th. We're using it at mozilla to do sentiment analysis, and we have 2
servers that stopped working at the same time.

(and by at the same time, I literally mean the last commited tweet was
the same: created_at: 2011-10-06T17:35:25.000Z )

I've seen 2 types of erros:

[2011-10-12 14:21:29,106][WARN ][river.twitter ] [Colleen
Wing] [twitter][twitter_river] stream failure, restarting stream...
Connection refusedTwitterException{exceptionCode=[39caf88f-15b6fe9c
39caf88f-15b6fe72], statusCode=-1, retryAfter=0, rateLimitStatus=null,
version=2.1.10}
at
twitter4j.internal.http.HttpClientImpl.request(HttpClientImpl.java:
325)
at
twitter4j.internal.http.HttpClientWrapper.request(HttpClientWrapper.java:
72)
at
twitter4j.internal.http.HttpClientWrapper.post(HttpClientWrapper.java:
103)
at twitter4j.TwitterStream.getFilterStream(TwitterStream.java:459)
at twitter4j.TwitterStream$8.getStream(TwitterStream.java:440)
at twitter4j.TwitterStream$8.getStream(TwitterStream.java:438)
at twitter4j.TwitterStream
$TwitterStreamConsumer.run(TwitterStream.java:673)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:
195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at sun.net.NetworkClient.doConnect(NetworkClient.java:158)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:394)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)
at sun.net.www.http.HttpClient.(HttpClient.java:233)
at sun.net.www.http.HttpClient.New(HttpClient.java:306)
at sun.net.www.http.HttpClient.New(HttpClient.java:323)
at
sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:
975)
at
sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:
916)
at
sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:
841)
at
sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:
1019)
at
twitter4j.internal.http.HttpClientImpl.request(HttpClientImpl.java:
283)
... 6 more
^C

and

[2011-10-12 23:07:05,273][WARN ][http.netty ] [Gyrich,
Henry Peter] Caught exception while handling client http traffic,
closing connection
java.io.IOException: Connection timed out
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:202)
at sun.nio.ch.IOUtil.read(IOUtil.java:169)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:
243)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:
321)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:
280)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
200)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

Any tips before I turn on debug mode? I'm doing some tests with the
latest snapshot version of twitter4j, but if anyone has tips on this
would be helpful

-pedro

Bowen_Masco · October 12, 2011, 9:51pm

I am not using the twitter river plugin, but I did had a similar
problem with a different twitter app.

It was caused by twitter's switch to ssl only( http to https urls)

https://dev.twitter.com/blog/streaming-api-turning-ssl-only-september-29th

The fix just required changing all the http urls to https urls for twitter.

Bowen

pedro · October 12, 2011, 10:00pm

I'll look into the twitter river code to see what's currently using, thanks
for the tip

On Wednesday, October 12, 2011, Bowen Masco bowen@codingfoo.com wrote:

I am not using the twitter river plugin, but I did had a similar
problem with a different twitter app.

It was caused by twitter's switch to ssl only( http to https urls)

https://dev.twitter.com/blog/streaming-api-turning-ssl-only-september-29th

The fix just required changing all the http urls to https urls for
twitter.

Bowen

Karussell1 · October 13, 2011, 7:32am

The twitter4j needs an update as I pointed out (on twitter).

created an issue here

On 13 Okt., 00:00, Pedro Alves pmgal...@gmail.com wrote:

I'll look into the twitter river code to see what's currently using, thanks
for the tip

On Wednesday, October 12, 2011, Bowen Masco bo...@codingfoo.com wrote:

I am not using the twitter river plugin, but I did had a similar
problem with a different twitter app.

It was caused by twitter's switch to ssl only( http to https urls)

https://dev.twitter.com/blog/streaming-api-turning-ssl-only-september...

The fix just required changing all the http urls to https urls for
twitter.

Bowen

pedro · October 13, 2011, 9:41am

Perfect!

I'll try again, just replacing the jar didn't work (even though it was
2am, don't even remember what I did). Also didn't find any classpath
references to the libs, but can also try to recompile from source to
see if it works

Thanks, I'll let know how it goes

On Thu Oct 13 08:32:29 2011, Karussell wrote:

The twitter4j needs an update as I pointed out (on twitter).

created an issue here

Twitter4j needs an update · Issue #1393 · elastic/elasticsearch · GitHub

On 13 Okt., 00:00, Pedro Alvespmgal...@gmail.com wrote:

I'll look into the twitter river code to see what's currently using, thanks
for the tip

On Wednesday, October 12, 2011, Bowen Mascobo...@codingfoo.com wrote:

I am not using the twitter river plugin, but I did had a similar
problem with a different twitter app.

It was caused by twitter's switch to ssl only( http to https urls)

https://dev.twitter.com/blog/streaming-api-turning-ssl-only-september...

The fix just required changing all the http urls to https urls for
twitter.

Bowen

pedro · October 13, 2011, 4:03pm

Got it. - thanks.

Created pull request to fix the issue

On Thu Oct 13 10:41:51 2011, Pedro Alves wrote:

Perfect!

I'll try again, just replacing the jar didn't work (even though it was
2am, don't even remember what I did). Also didn't find any classpath
references to the libs, but can also try to recompile from source to
see if it works

Thanks, I'll let know how it goes

On Thu Oct 13 08:32:29 2011, Karussell wrote:

The twitter4j needs an update as I pointed out (on twitter).

created an issue here

Twitter4j needs an update · Issue #1393 · elastic/elasticsearch · GitHub

On 13 Okt., 00:00, Pedro Alvespmgal...@gmail.com wrote:

I'll look into the twitter river code to see what's currently using,
thanks
for the tip

On Wednesday, October 12, 2011, Bowen Mascobo...@codingfoo.com wrote:

I am not using the twitter river plugin, but I did had a similar
problem with a different twitter app.

It was caused by twitter's switch to ssl only( http to https urls)

https://dev.twitter.com/blog/streaming-api-turning-ssl-only-september...

The fix just required changing all the http urls to https urls for
twitter.

Bowen