Cloudwatch Logs to Elasticsearch using AWS Lambda

Elastic Stack Elasticsearch
1

Hi everyone, I am trying to confirm my tehcnical approach on solving a problem.

Background Information:

  • Elasticsearch: 7.0.1
  • Logstash: 7.0.1
  • Kibana: 7.0.1
  • I need a serverless solution on how to transfer AWS Cloudwatch logs to Kibana.
  • I am using EISMonitoringSystem which involves (Elasticsearch, Logstash, Kibanna, and possibly Beats).
  • I have also AWS Cloudwatch logs which I need to transfer to Kibana for visualization.
    Steps include:

Cloudwatch Logs > Logstash > Elasticsearch > Kibana

Question
1. I know that you can run Logstash and configure its input plugin to receive cloudwatch logs, but this only works if Logstash is running my machine locally right? How can I make Cloudwatch transfer automatically once they are uploaded in AWS (I don't want to run anything on my machine to do this, I want it to be automatic)?

2. I read online about an approach involving Kinesis Firehose Stream to transfer Cloudwatch logs to Elasticsearch. The steps where as followed?

AWS Config → S3 → 𝝺 load_from_s3 → 𝝺 transform_data → 𝝺 send_to_kinesis_stream → Kinesis Firehose → ES

Does the approach above going to be automatic meaning logs transfer to Elasticsearch (not on localhost, the one offered by Elastic online) automatically?

Feel free to give your suggestions, any information helps. Thanks!

2

Hi,

You can check this project: https://github.com/DataDog/datadog-serverless-functions/tree/master/aws/logs_monitoring

From the discussion here: Add a field to json body when indexing

1 Like
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.