How to stream log data from the Kinesis stream to Elasticsearch?

Elastic Stack Logstash
1

How to stream log data from the Kinesis stream to Elasticsearch?

I have used logstash kinesis input plugin, but it doesn't seems to works.

This is my logstash configuration for the Kinesis stream.

input {
 kinesis {
    application_name => "test-kinesis"
    kinesis_stream_name => "test-kinesis"
    region => "ap-southeast-1"
    profile => "test-kinesis"
    initial_position_in_stream => "TRIM_HORIZON"
    codec => cloudwatch_logs
    id => "kinesis-rq-log-in"
 }
}

output {
 elasticsearch {
  hosts => ['test-kinesis-ip:9200', 'test-kinesis2-ip:9200']
  user => "root"
  password => "p@ssword"
  timeout => 60
  ssl => true
  cacert => '/etc/elasticsearch/x-pack/trusted-cert/ca.cert.pem'
  manage_template => false
  index => "kinesis-rq-%{[@metadata][localtime]}"
  document_type => "http"
  id => "kinesis-rq-log-out"
 }
}

I tried to follow the plugin document, but I still could not get the log data on Kibana with the index pattern "kinesis-rq*".

Also, have tried to solve that following with "https://github.com/logstash-plugins/logstash-input-kinesis/issues/20", but not found the useful solution.

Does anyone have any idea?

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.