LS pipeline start but no ES index created [SOLVED]

Elastic Stack Logstash
1

Newbie here. The first time I ran through the LS guide, I was able to start a pipeline that created an index. However, I have not been able to start a pipeline that created an index since. I'm thinking that maybe the port is incorrect?

Here is my config file:

input {
  twitter {
      consumer_key => "key"
      consumer_secret => "secret"
      oauth_token => "oauth"
      oauth_token_secret => "authsecret"
      keywords => [ "testtesttest" ]
      full_tweet => true
  }
}

output {
  stdout { codec => dots }
  elasticsearch {
    action => "index"
    index => "twitter1"
    document_type => "tweet"
    template => "twitter_template.json"
    template_name => "twitter"
  }
}

Here is some debug logs that I think may be relevant:

[2016-11-02T16:08:26,353][DEBUG][org.apache.http.impl.conn.PoolingHttpClientConnectionManager] Connection released: [id: 0][route: {}->http://127.0.0.1:9200][total kept alive: 1; route allocated: 1 of 100; total allocated: 1 of 1000]
[2016-11-02T16:08:26,354][DEBUG][logstash.outputs.elasticsearch] Found existing Elasticsearch template. Skipping template management {:name=>"twitter"}
[2016-11-02T16:08:26,355][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["127.0.0.1"]}
[2016-11-02T16:08:26,364][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>1, "pipeline.batch.delay"=>60, "pipeline.max_inflight"=>4}
[2016-11-02T16:08:26,395][INFO ][logstash.pipeline        ] Pipeline main started
[2016-11-02T16:08:26,407][DEBUG][logstash.agent           ] Starting puma
[2016-11-02T16:08:26,472][DEBUG][logstash.api.service     ] [api-service] start
[2016-11-02T16:08:26,474][DEBUG][logstash.agent           ] Trying to start WebServer {:port=>9612}
[2016-11-02T16:08:26,474][DEBUG][logstash.api.service     ] [api-service] start
[2016-11-02T16:08:26,481][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9612}

Any insight would be very helpful!

2

is the twitter input producing any tweets? the index will only be created with the first event

3

I've verified there are tweets using the string I am searching for. However, the pipeline is not pulling any of these in. I tried again this morning and still no dice. :confused:

Here is an excerpt from the logs this morning:

[2016-11-03T11:03:47,644][DEBUG][org.apache.http.impl.execchain.MainClientExec] Connection can be kept alive indefinitely
[2016-11-03T11:03:47,644][DEBUG][org.apache.http.impl.conn.PoolingHttpClientConnectionManager] Connection [id: 0][route: {}->http://127.0.0.1:9200] can be kept alive indefinitely
[2016-11-03T11:03:47,644][DEBUG][org.apache.http.impl.conn.PoolingHttpClientConnectionManager] Connection released: [id: 0][route: {}->http://127.0.0.1:9200][total kept alive: 1; route allocated: 1 of 100; total allocated: 1 of 1000]
[2016-11-03T11:03:47,645][DEBUG][logstash.outputs.elasticsearch] Found existing Elasticsearch template. Skipping template management {:name=>"twitter"}
[2016-11-03T11:03:47,646][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["127.0.0.1"]}
[2016-11-03T11:03:47,656][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>1, "pipeline.batch.delay"=>60, "pipeline.max_inflight"=>4}
[2016-11-03T11:03:47,659][INFO ][logstash.pipeline        ] Pipeline main started
[2016-11-03T11:03:47,671][DEBUG][logstash.agent           ] Starting puma
[2016-11-03T11:03:47,672][DEBUG][logstash.agent           ] Trying to start WebServer {:port=>9600}
[2016-11-03T11:03:47,673][DEBUG][logstash.api.service     ] [api-service] start
[2016-11-03T11:03:47,710][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2016-11-03T11:03:47,793][DEBUG][logstash.instrument.collector] Collector: Sending snapshot to observers {:created_at=>2016-11-03 11:03:47 -0700}
[2016-11-03T11:03:48,798][DEBUG][logstash.instrument.collector] Collector: Sending snapshot to observers {:created_at=>2016-11-03 11:03:48 -0700}
[2016-11-03T11:03:49,805][DEBUG][logstash.instrument.collector] Collector: Sending snapshot to observers {:created_at=>2016-11-03 11:03:49 -0700}
[2016-11-03T11:03:50,809][DEBUG][logstash.instrument.collector] Collector: Sending snapshot to observers {:created_at=>2016-11-03 11:03:50 -0700}
[2016-11-03T11:03:51,823][DEBUG][logstash.instrument.collector] Collector: Sending snapshot to observers {:created_at=>2016-11-03 11:03:51 -0700}
[2016-11-03T11:03:52,662][DEBUG][logstash.pipeline        ] Pushing flush onto pipeline
[2016-11-03T11:03:52,829][DEBUG][logstash.instrument.collector] Collector: Sending snapshot to observers {:created_at=>2016-11-03 11:03:52 -0700}
[2016-11-03T11:03:53,836][DEBUG][logstash.instrument.collector] Collector: Sending snapshot to observers {:created_at=>2016-11-03 11:03:53 -0700}
[2016-11-03T11:03:54,841][DEBUG][logstash.instrument.collector] Collector: Sending snapshot to observers {:created_at=>2016-11-03 11:03:54 -0700}
[2016-11-03T11:03:55,845][DEBUG][logstash.instrument.collector] Collector: Sending snapshot to observers {:created_at=>2016-11-03 11:03:55 -0700}
[2016-11-03T11:03:56,848][DEBUG][logstash.instrument.collector] Collector: Sending snapshot to observers {:created_at=>2016-11-03 11:03:56 -0700}
4

I got it working. Thank you João.

I realized the account that I was sending tweets from was set to private haha.

1 Like