Unable to pull from kineses using the kinesis logstash plugin

Hi
I have installed logstash 7.1.1 on centos 7, I have installed the kineseis plugin and codec plugin. the aws credentials and the perm to access the stream are all good what i see in the logs is and shows its connected to my stream
Jun 12 05:38:43 logstash-server logstash: OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Jun 12 05:39:23 logstash-server logstash: Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
Jun 12 05:39:24 logstash-server logstash: [2019-06-12T05:39:24,759][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.1.1"}
Jun 12 05:39:37 logstash-server logstash: [2019-06-12T05:39:37,105][INFO ][logstash.javapipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>125, :thread=>"#<Thread:0x550e226f run>"}
Jun 12 05:39:37 logstash-server logstash: [2019-06-12T05:39:37,643][INFO ][logstash.javapipeline ] Pipeline started {"pipeline.id"=>"main"}
Jun 12 05:39:37 logstash-server logstash: [2019-06-12T05:39:37,818][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
Jun 12 05:39:37 logstash-server logstash: Jun 12, 2019 5:39:37 AM com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker createClient
Jun 12 05:39:37 logstash-server logstash: WARNING: Received configuration for region as us-west-2.
Jun 12 05:39:38 logstash-server logstash: Jun 12, 2019 5:39:38 AM com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker createClient
Jun 12 05:39:38 logstash-server logstash: WARNING: Received configuration for region as us-west-2.
Jun 12 05:39:38 logstash-server logstash: Jun 12, 2019 5:39:38 AM com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker createClient
Jun 12 05:39:38 logstash-server logstash: WARNING: Received configuration for region as us-west-2.
Jun 12 05:39:39 logstash-server logstash: [2019-06-12T05:39:39,474][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

As a test i have posted data to my stream using json and also verified pulling it using a python program and all works, however with this configuration

input {
kinesis {
kinesis_stream_name => "estest"
application_name => "logstash"
profile => "xxxx"
initial_position_in_stream => "TRIM_HORIZON"
region => "xxxx"

}
}

output {
file {
path => "/tmp/estest"

}
}

With the above config i dont see anything happening, it doesnt seem to pull the messages.
Plz advise any bug in the plugin or 7.1.1 logstash?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.