Twitter input

Elastic Stack Logstash
1

this is my code logastash.conf. The problem I have is that there is no index data

input {
          twitter { 
         consumer_key => "dplzZiWV2qP5q2dMZpM8lLb0c"
          consumer_secret =>"wpemcks8EqdgY3ZjV8dMJOViB0p                               QuZsWOPtnH65jHAJdvyyaDW"
     oauth_token => "738404980470730752-     lrULPA0EEPZHlMDOyDNVoSFdfvykyKJ"
     oauth_token_secret =>     "ydtfWx6QZ5a5CTG1R2rA7wCTcBwBVElvOfdVP9Uxo3JE3"
     keywords => [ "tecnologia" ]
     full_tweet => true
}
}

filter {
  }

 output {
 stdout { codec => dots }
 elasticsearch {
 protocol => ["http"]
 hosts => ["localhost:9200"]
 index => "twitter"
 document_type => "tweet"
  }
}
2

in the fields they appear Kibana me but tell me not index

kibana.JPG962×640 83.6 KB

3

this data gives me logstash

     "created_at" => "Tue Jun 28 17:35:37 +0000 2016",
                   "id" => 747845925133967361,
               "id_str" => "747845925133967361",
                 "text" => "El Sol parece adentrarse en una nueva fase evolutiva        https://t.co/US9CDau5Na #tecnologia",
                   "source" => "<a href=\"http://twitterfeed.com\"   rel=\"nofollow\">twitterfeed</a>",
            "truncated" => false,
"in_reply_to_status_id" => nil,
"in_reply_to_status_id_str" => nil,
  "in_reply_to_user_id" => nil,
  "in_reply_to_user_id_str" => nil,
"in_reply_to_screen_name" => nil,
                 "user" => {
                                "id" => 1678983510,
                            "id_str" => "1678983510",
                              "name" => "Tecnoticias",
                       "screen_name" => "tuitTecnologia",
                          "location" => "Madrid",
                               "url" => nil,
                       "description" => "Padre de @tuitApple y @tuitAndroid.       Publicamos cualquier #noticia sobre #ciencia y #tecnología: #móviles,    #pcs, #consolas, #internet, #apple, #android y más.",
                         "protected" => false,
                          "verified" => false,
                   "followers_count" => 7588,
                     "friends_count" => 6400,
                      "listed_count" => 202,
                  "favourites_count" => 23,
                    "statuses_count" => 245511,
                        "created_at" => "Sat Aug 17 19:50:14 +0000 2013",
                        "utc_offset" => -14400,
                         "time_zone" => "Eastern Time (US & Canada)",
                       "geo_enabled" => false,
                              "lang" => "es",
              "contributors_enabled" => false,
                     "is_translator" => false,
          "profile_background_color" => "1A1B1F",
      "profile_background_image_url" =>      "http://abs.twimg.com/images/themes/theme9/bg.gif",
"profile_background_image_url_https" =>         "https://abs.twimg.com/images/themes/theme9/bg.gif",
           "profile_background_tile" => false,
                "profile_link_color" => "2F39ED",
      "profile_sidebar_border_color" => "FFFFFF",
        "profile_sidebar_fill_color" => "DDEEF6",
                "profile_text_color" => "333333",
      "profile_use_background_image" => true,
                 "profile_image_url" => "http://pbs.twimg.com/profile_images/378800000391393974/5806f3428f11d4cfc6fd4de4ad28b9c3_normal.jpeg",
           "profile_image_url_https" => "https://pbs.twimg.com/profile_images/378800000391393974/5806f3428f11d4cfc6fd4de4ad28b9c3_normal.jpeg",
                "profile_banner_url" => "https://pbs.twimg.com/profile_banners/1678983510/1406578558",
                   "default_profile" => false,
             "default_profile_image" => false,
                         "following" => nil,
               "follow_request_sent" => nil,
                     "notifications" => nil
     },
                  "geo" => nil,
          "coordinates" => nil,
                "place" => nil,
         "contributors" => nil,
      "is_quote_status" => false,
        "retweet_count" => 0,
       "favorite_count" => 0,
             "entities" => {
     "hashtags" => [
    [0] {
           "text" => "tecnologia",
        "indices" => [
            [0] 77,
            [1] 88
        ]
    }
],
         "urls" => [
    [0] {
                 "url" => "https://t.co/US9CDau5Na",
        "expanded_url" => "http://tecnologia.tusueldo.com/14Ol",
         "display_url" => "tecnologia.tusueldo.com/14Ol",
             "indices" => [
            [0] 53,
            [1] 76
        ]
    }
],
"user_mentions" => [],
      "symbols" => []
 },
            "favorited" => false,
            "retweeted" => false,
   "possibly_sensitive" => false,
         "filter_level" => "low",
                 "lang" => "es",
         "timestamp_ms" => "1467135337514",
             "@version" => "1",
           "@timestamp" => "2016-06-28T17:35:37.000Z"
4

What version of Logstash are you using? protocol is not a supported option on recent versions. See Elasticsearch output plugin | Logstash Reference [8.11] | Elastic

5

hi Aaron
I'm using version 2.3.1

6

The configuration as you've given should not even be able to run. Here below, I tested a simple example with similar configuration parameters (including protocol):

input {
  stdin {}
}

filter {}

output {
  elasticsearch {
    hosts => ["127.0.0.1:9200"]
    index => "dummy"
    document_type => "foo"
    protocol => ["http"]
  }
}

And when I try to run it, this is the output:

$ bin/logstash -f ~/logstash-test.conf --verbose
starting agent {:level=>:info}
starting pipeline {:id=>"main", :level=>:info}
Settings: Default pipeline workers: 8
Unknown setting 'protocol' for elasticsearch {:level=>:error}
Pipeline aborted due to error {:exception=>#<LogStash::ConfigurationError: Something is wrong with your configuration.>, :backtrace=>["/opt/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/config/mixin.rb:134:in `config_init'", "/opt/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/outputs/base.rb:63:in `initialize'", "/opt/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/output_delegator.rb:74:in `register'", "/opt/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:181:in `start_workers'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:181:in `start_workers'", "/opt/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:136:in `run'", "/opt/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/agent.rb:465:in `start_pipeline'"], :level=>:error}
stopping pipeline {:id=>"main"}
Closing inputs {:level=>:info}
Closed inputs {:level=>:info}

Please note the error message: Unknown setting 'protocol' for elasticsearch {:level=>:error}

7

I do not take the http protocol and provides us with elasticsearch_java plugin, and see the same Kibana

8

I highly recommend you switch to using the standard elasticsearch output plugin soon, as there is a strong possibility that the node protocol will be removed soon. The transport protocol may also go away in future versions.

That said, if you're having Kibana configuration issues, you should take those questions to https://discuss.elastic.co/c/kibana

9

the fields are empty as " geo" = > nil in logstash may be because it takes a long time to take that data, and for that reason logstash passes empty , they said I could use REST API but not how to use it. You know if you can use?

10

Are all of the geo fields empty, or just some? I mean, it seems possible that some tweets just don't have that information because of how they were submitted, or from where (no reverse lookup of the location possible).

11

no data in any field geo

12

You should run with --debug turned on and see what comes up.

It may be that Twitter isn't sending the geo field. It may be that something else is going on.

13

ok thank you

14

It shows me this error and is not referred

 Twitter too many requests error, sleeping for s {:level=>:warn, :file=>"logstash/inputs/twitter.rb", :line=>"134", :method=>"run"}
15

Well, that suggests why you aren't seeing data: Twitter is throttling you.