I'm using terraform to do the deployment.
I did as follows:
provider "aws" {
region = "us-east-1" ## FILL WITH THE AWS REGION WHERE YOU WANT TO DEPLOY THE ELASTIC SERVERLESS FORWARDER
}
data "aws_serverlessapplicationrepository_application" "esf_sar" {
application_id = "arn:aws:serverlessrepo:eu-central-1:267093732750:applications/elastic-serverless-forwarder"
}
resource "aws_serverlessapplicationrepository_cloudformation_stack" "esf_cf_stak" {
name = "terraform-elastic-serverless-forwarder"
application_id = data.aws_serverlessapplicationrepository_application.esf_sar.application_id
semantic_version = data.aws_serverlessapplicationrepository_application.esf_sar.semantic_version
capabilities = data.aws_serverlessapplicationrepository_application.esf_sar.required_capabilities
parameters = {
ElasticServerlessForwarderS3ConfigFile = "s3://elk-config-sar-bucket-dev/sarconfig.yaml" ## FILL WITH THE VALUE OF THE S3 URL IN THE FORMAT "s3://bucket-name/config-file-name" POINTING TO THE CONFIGURATION FILE FOR YOUR DEPLOYMENT OF THE ELASTIC SERVERLESS FORWARDER
ElasticServerlessForwarderSSMSecrets = "" ## FILL WITH A COMMA DELIMITED LIST OF AWS SSM SECRETS ARNS REFERENCED IN THE CONFIG YAML FILE (IF ANY).
ElasticServerlessForwarderKMSKeys = "" ## FILL WITH A COMMA DELIMITED LIST OF AWS KMS KEYS ARNS TO BE USED FOR DECRYPTING AWS SSM SECRETS REFERENCED IN THE CONFIG YAML FILE (IF ANY).
ElasticServerlessForwarderSQSEvents = "" ## FILL WITH A COMMA DELIMITED LIST OF DIRECT SQS QUEUES ARNS TO SET AS EVENT TRIGGERS FOR THE LAMBDA (IF ANY).
ElasticServerlessForwarderS3SQSEvents = "" ## FILL WITH A COMMA DELIMITED LIST OF S3 SQS EVENT NOTIFICATIONS ARNS TO SET AS EVENT TRIGGERS FOR THE LAMBDA (IF ANY).
ElasticServerlessForwarderKinesisEvents = "" ## FILL WITH A COMMA DELIMITED LIST OF KINESIS DATA STREAM ARNS TO SET AS EVENT TRIGGERS FOR THE LAMBDA (IF ANY).
ElasticServerlessForwarderCloudWatchLogsEvents = "arn:aws:logs:us-east-1:*****45454:log-group:/aws/lambda/pod-loan-asyc-dev-check-expired-order:*" ## FILL WITH A COMMA DELIMITED LIST OF CLOUDWATCH LOGS LOG GROUPS ARNS TO SET SUBSCRIPTION FILTERS ON THE LAMBDA FOR (IF ANY).
ElasticServerlessForwarderS3Buckets = "" ## FILL WITH A COMMA DELIMITED LIST OF S3 BUCKETS ARNS THAT ARE THE SOURCES OF THE S3 SQS EVENT NOTIFICATIONS (IF ANY).
}
}
My sarconfig.yaml
inputs:
- type: "cloudwatch-logs"
id: "arn:aws:logs:us-east-1:*****45454:log-group:/aws/lambda/pod-loan-asyc-dev-check-expired-order:*"
outputs:
- type: "elasticsearch"
args:
# either elasticsearch_url or cloud_id, elasticsearch_url takes precedence
#elasticsearch_url: "http(s)://domain.tld:port"
cloud_id: "AWS_test:d********************pbyRmZjBmNDBlNTVmMzY0MWYwYWI2NTg5ODNmN2E4ODllNCRlZmE5NDQ1NWZlZmI0YWIxYWYzOGQ4ZmQ5MDFkOTY3Yw=="
# either api_key or username/password, api_key takes precedence
api_key: "******************Tk86WDIxWXg1X1FSaDJfRlJlQnc2c2QyQQ=="
#username: "username"
#password: "password"
es_datastream_name: "test"
batch_max_actions: 500
batch_max_bytes: 10485760
Permissions are like this, it was added by deployment, I didn't change it
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"logs:DescribeLogGroups",
"s3:GetObject",
"sqs:SendMessage"
],
"Resource": [
"arn:aws:logs:us-east-1:*****45454:log-group:/aws/lambda/condo-loan-asyc-dev-check-expired-order:*",
"arn:aws:s3:::elk-config-sar-bucket-dev/sarconfig.yaml",
"arn:aws:sqs:us-east-1:*****45454:elastic-serverless-forwarder-replay-queue-128e6a1cd335",
"arn:aws:sqs:us-east-1:*****45454:elastic-serverless-forwarder-continuing-queue-128e6a1cd335"
]
}
]
}
I did a test and released it with permission for everyone:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"sqs:SendMessage"
],
"Resource": [
"arn:aws:s3:::elk-config-sar-bucket-dev/sarconfig.yaml",
"arn:aws:sqs:us-east-1:*****45454:elastic-serverless-forwarder-replay-queue-128e6a1cd335",
"arn:aws:sqs:us-east-1:*****45454:elastic-serverless-forwarder-continuing-queue-128e6a1cd335"
]
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": "logs:*",
"Resource": [
"arn:aws:logs:us-east-1:*****45454:log-group:/aws/lambda/pod-loan-asyc-dev-check-expired-order:*",
"arn:aws:logs:*:*****45454:log-group:*:log-stream:*",
"arn:aws:logs:*:*****45454:destination:*",
"arn:aws:logs:*:*****45454:log-group:*"
]
}
]
}
Now we have no error log, but no input yet in Elastic.
{
"@timestamp": "2022-06-24T06:31:54.515Z",
"log.level": "warning",
"message": "no input defined",
"ecs": {
"version": "1.6.0"
},
"input_id": "arn:aws:logs:us-east-1:*****45454:log-group:/aws/lambda/pod-loan-asyc-dev-check-expired-order:*",
"input_type": "cloudwatch-logs",
"log": {
"logger": "root",
"origin": {
"file": {
"line": 119,
"name": "handler.py"
},
"function": "lambda_handler"
},
"original": "no input defined"
},
"process": {
"name": "MainProcess",
"pid": 9,
"thread": {
"id": 139822395262784,
"name": "MainThread"
}
}
}
