I am using EC2 Instance(Linux) and Logstash v2.3.4 . Logstash does not support directly writing compressed data to kinesis stream.One of the updates to input configuration that I have made is to read input data from S3 and store compressed data to local folder (EC2 Instance). Another input configuration is related to reading compressed data from local folder (EC2 Instance) and write data to kinesis stream.I am facing issue in which I am unable to read compressed data from kinesis stream. Kinesis dashboard display traffic information but when I am trying to read data from kinesis it returns null. I am using following python script for reading data from kinesis stream:
from boto import kinesis
import timekinesis = kinesis.connect_to_region("region in which my kinesis stream exist")
shard_id = 'shardId-000000000000' #My stream have only one shard!
shard_it = kinesis.get_shard_iterator("myKinesisStreamName", shard_id, "LATEST")["ShardIterator"]
while 1==1:
... out = kinesis.get_records(shard_it, limit=2)
... shard_it = out["NextShardIterator"]
... print out;
... time.sleep(0.2)
...