How to use twitter plugin in Logstash 5.x

Elastic Stack Logstash
1

Hi

I want to pull tweets matching particular keywords into ELK, adding a field to show what was matched. I can do this, but solution does not scale and I think there is likely a better way.

Here's my conf files, I think intent is self explanatory.

input {  
    twitter {
        # add your data
        consumer_key => "xx"
        consumer_secret => "xx"
        oauth_token => "xx-xx"
        oauth_token_secret => "xx"
        keywords => ["london"]
        full_tweet => true
        add_field => { "myplace" => "london" }
    }
}
input {  
    twitter {
        # add your data
        consumer_key => "xx"
        consumer_secret => "xx"
        oauth_token => "xx-xx"
        oauth_token_secret => "xx"
        keywords => ["paris"]
        full_tweet => true
        add_field => { "myplace" => "paris" }
    }
}
output {  
    elasticsearch {
       index => "twitter-%{+YYYY.MM.dd}"
    }
}

I was using if condition on output to output into different indices, but that has same problem.

Once I get over 2 or so input sections, my logs fill up with

[WARN ][logstash.inputs.twitter ] Twitter too many requests error, sleeping for 300s

which is a message I understand but

  1. I don't see how I can control/stop this from my end

and

  1. I don't see a better way than the above.

Thanks for suggestions,
RT

2

Hi

er, anyone have any ideas on this?

RT

3

Maybe you need to register 2 different Twitter applications, one for each input.

4

I tried that. But API limits are per account, and not per app.

5

Sorry for not being of any help.

6

Thank you for trying!!

I have updated to 5.2 today but same behavior.

7

Hard luck! Not sure, but I am thinking about something like this:

input {
twitter {
# add your data
consumer_key => "xx"
consumer_secret => "xx"
oauth_token => "xx-xx"
oauth_token_secret => "xx"
keywords => ["london","paris"]
full_tweet => true
}
}
filter {
mutate {add_field => { "myplace" => "london"}
}
if ["paris"] in ["text"] {
update => { "myplace" => "paris" }
}
}
output {
stdout {
codec => dots
}
elasticsearch {
hosts => "http://localhost:9200"
index => "twitter-%{+YYYY.MM.dd}"
document_type => "tweet"
template => "twitter_template.json"
template_name => "twitter-*"
template_overwrite => true
}
}

Only one input, "london" in "myplace" from the beginning, and update it to "paris" when this keyword is in the "text" field of the tweet. You also need a template.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.