Hello,
At the moment i have logstash and elasticsearch. My logstash uses as input the twitter input plug in and as output i have the elasticsearch plugin. My goal is to invoke a python script before the data is stored in my elasticsearch. Now the script is something simple like adding a new field with static text just to see it working but later on it will do some text/sentiment analysis. So i dont want to use the ruby filter. I dont get any errors i can see the twitter posts in my elasticsearch but without the new field that i expect. (i also made the script write to file just to see if it is invoked an nothing). I have installed the python plugin in elasticsearch and my script is inside the config/scripts module. My logstash configuration so far is
input {
twitter{
consumer_key => ""
consumer_secret => ""
oauth_token => ""
oauth_token_secret => ""
keywords => ["webinar"]
languages => ["en"]
full_tweet => true
}
}
filter{
if "webinar" not in [text]{
drop { }
}
if [user][lang]{
ruby {
code => 'event.set("language", event.get("[user][lang]"))'
}
}
if [user][time_zone]{
ruby {
code => 'event.set("timezone", event.get("[user][time_zone]"))'
}
}
if [user][location]{
ruby {
code => 'event.set("location", event.get("[user][location]"))'
}
}
prune {
whitelist_names => ["^created_at$","^retweet_count$","^text$","^language$","^timezone$","^location$"]
}
}
output {
stdout{codec => rubydebug}
elasticsearch {
hosts => ["localhost:9200"]
index => "twitterposts"
script_lang => "python"
script_type => "file"
script => "latlongcalc.py"
}
}
Also if there is a smarter way to store specific nested fields from user without coping the value to a new field that would be also nice.
Thank you in advance.