Twitter client error: Gateway Timeout

[pastechecker (Charlie)]

Hello,
I am getting following error:

[2018-09-06T14:26:15,801][WARN ][logstash.inputs.twitter ] Twitter client error {:message=>"", :exception=>"Twitter::Error::GatewayTimeout",
:backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/twitter-5.15.0/lib/twitter/streaming/response.rb:21:in `on_headers_complete'",
"org/ruby_http_parser/RubyHttpParser.java:370:in `<<'",
"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/twitter-5.15.0/lib/twitter/streaming/response.rb:16:in `<<'",
"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-twitter-3.0.8/lib/logstash/inputs/twitter/patches.rb:31:in `stream'",
"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-twitter-3.0.8/lib/logstash/inputs/twitter/patches.rb:58:in `request'",
"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/twitter-5.15.0/lib/twitter/streaming/client.rb:37:in `filter'",
"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-twitter-3.0.8/lib/logstash/inputs/twitter.rb:138:in `run'",
"/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:408:in `inputworker'",
"/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:402:in `block in start_input'"], :options=>nil}

tcpdump snippet:

14:29:28.656778 Flags [P.], seq 1:2448, ack 1, win 229, options [nop,nop,TS val 3071903 ecr 3639309316], length 2447: HTTP: POST https://stream.twitter.com/1xthe_rest_of_the_request.
14:30:07.316802 Flags [FP.], seq 1:375, ack 2448, win 32462, options [nop,nop,TS val 3859320081 ecr 3062234], length 374: HTTP: HTTP/1.1 504 Gateway Timeout

Config:

input {
twitter {
consumer_key => "XXX"
consumer_secret => "XXX"
oauth_token => "XXX"
oauth_token_secret => "XXX"
full_tweet => true
ignore_retweets => false
follows => ["962950647812448261","848316160"]
use_samples => false
languages => ["en", "de"]
use_proxy => true
proxy_address => "172.11.54.10"
proxy_port => "8080"
}
}

filter {}

output {
stdout { codec => dots }
elasticsearch {
hosts => ["localhost:9200"]
index => "twitter"
document_type => "tweets"
template_overwrite => true
template => "/etc/logstash/conf.d/backupconfig/twitter_template.json"
}
}

The requests are behind a reverse proxy.
Is there a way to increase gateway timeout?

[yaauie (Ry Biesemeyer)]

The HTTP: HTTP/1.1 504 Gateway Timeout in your tcpdump indicates to me that your reverse proxy is severing the connection and sending the 504.

If this is the case, increasing the timeouts on your reverse proxy should improve the situation.

It is also possible that your reverse proxy is unable to establish a connection to the remote host. Checking the logs of your reverse proxy should provide hints.

[pastechecker (Charlie)]

Thanks.