How to integrate AWS Lambda with plugin Elastic

Hello @Renato_Souza

the ARNs have to be defined in both places: when deploying the Forwarder, so that proper permissions and triggers are set, and in the config file, so that you can define the output of a given input ARN

it's suggested to create and upload the config file to an s3 bucket, before deploying the Forwarder.
Also if you update the deployment, it is suggested to update the config file before, at least for adding new input ARNs

if you need to update the outputs only, you just update the config file and next execution of the lambda will use the updated config, with no needs for the Forwarder to be redeployed

okay, I successfully implanted. But there is some permission adjustment required, see:

That is right?

did you add the cloudwatch logs arn to the ElasticServerlessForwarderCloudWatchLogsEvents param when deploying the forwarder?

if you want to add an input after the first deployment you have to redeploy the forwarder: for that you follow the same step as creating a new function using the same "Application name". this will update the already deployed forwarder instead of creating a new one

beware that due to a bug in AWS CloudFormation, if you want to update the Events settings for the deployed Lambda, you will have to execute a deployment deleting the existing ones before actually apply the new updated ones.

this means that you have to update it twice: the first time emptying the following params:

      ElasticServerlessForwarderCloudWatchLogsEvents
      ElasticServerlessForwarderKinesisEvents
      ElasticServerlessForwarderS3SQSEvents
      ElasticServerlessForwarderSQSEvents

and the second time filling them again with the new values

if you updated the deployment adding the new value for ElasticServerlessForwarderCloudWatchLogsEvents without emptying it before, the deployment should have failed.

you can check in the cloudformation console on AWS or also see in the lambda details for the forwarder what are the permissions and the triggers set up for the lambda: the ones added with the update should not be present

I'm using terraform to do the deployment.

I did as follows:

provider "aws" {
  region = "us-east-1"  ## FILL WITH THE AWS REGION WHERE YOU WANT TO DEPLOY THE ELASTIC SERVERLESS FORWARDER
}

data "aws_serverlessapplicationrepository_application" "esf_sar" {
  application_id = "arn:aws:serverlessrepo:eu-central-1:267093732750:applications/elastic-serverless-forwarder"
}

resource "aws_serverlessapplicationrepository_cloudformation_stack" "esf_cf_stak" {
  name             = "terraform-elastic-serverless-forwarder"
  application_id   = data.aws_serverlessapplicationrepository_application.esf_sar.application_id
  semantic_version = data.aws_serverlessapplicationrepository_application.esf_sar.semantic_version
  capabilities     = data.aws_serverlessapplicationrepository_application.esf_sar.required_capabilities

  parameters = {
    ElasticServerlessForwarderS3ConfigFile         = "s3://elk-config-sar-bucket-dev/sarconfig.yaml"  ## FILL WITH THE VALUE OF THE S3 URL IN THE FORMAT "s3://bucket-name/config-file-name" POINTING TO THE CONFIGURATION FILE FOR YOUR DEPLOYMENT OF THE ELASTIC SERVERLESS FORWARDER
    ElasticServerlessForwarderSSMSecrets           = ""  ## FILL WITH A COMMA DELIMITED LIST OF AWS SSM SECRETS ARNS REFERENCED IN THE CONFIG YAML FILE (IF ANY).
    ElasticServerlessForwarderKMSKeys              = ""  ## FILL WITH A COMMA DELIMITED LIST OF AWS KMS KEYS ARNS TO BE USED FOR DECRYPTING AWS SSM SECRETS REFERENCED IN THE CONFIG YAML FILE (IF ANY).
    ElasticServerlessForwarderSQSEvents            = ""  ## FILL WITH A COMMA DELIMITED LIST OF DIRECT SQS QUEUES ARNS TO SET AS EVENT TRIGGERS FOR THE LAMBDA (IF ANY).
    ElasticServerlessForwarderS3SQSEvents          = ""  ## FILL WITH A COMMA DELIMITED LIST OF S3 SQS EVENT NOTIFICATIONS ARNS TO SET AS EVENT TRIGGERS FOR THE LAMBDA (IF ANY).
    ElasticServerlessForwarderKinesisEvents        = ""  ## FILL WITH A COMMA DELIMITED LIST OF KINESIS DATA STREAM ARNS TO SET AS EVENT TRIGGERS FOR THE LAMBDA (IF ANY).
    ElasticServerlessForwarderCloudWatchLogsEvents = "arn:aws:logs:us-east-1:*****45454:log-group:/aws/lambda/pod-loan-asyc-dev-check-expired-order:*"  ## FILL WITH A COMMA DELIMITED LIST OF CLOUDWATCH LOGS LOG GROUPS ARNS TO SET SUBSCRIPTION FILTERS ON THE LAMBDA FOR (IF ANY).
    ElasticServerlessForwarderS3Buckets            = ""  ## FILL WITH A COMMA DELIMITED LIST OF S3 BUCKETS ARNS THAT ARE THE SOURCES OF THE S3 SQS EVENT NOTIFICATIONS (IF ANY).
  }
}

My sarconfig.yaml

inputs:
  - type: "cloudwatch-logs"
    id: "arn:aws:logs:us-east-1:*****45454:log-group:/aws/lambda/pod-loan-asyc-dev-check-expired-order:*"
    outputs:
      - type: "elasticsearch"
        args:
          # either elasticsearch_url or cloud_id, elasticsearch_url takes precedence
          #elasticsearch_url: "http(s)://domain.tld:port"
          cloud_id: "AWS_test:d********************pbyRmZjBmNDBlNTVmMzY0MWYwYWI2NTg5ODNmN2E4ODllNCRlZmE5NDQ1NWZlZmI0YWIxYWYzOGQ4ZmQ5MDFkOTY3Yw=="
          # either api_key or username/password, api_key takes precedence
          api_key: "******************Tk86WDIxWXg1X1FSaDJfRlJlQnc2c2QyQQ=="
          #username: "username"
          #password: "password"
          es_datastream_name: "test"
          batch_max_actions: 500
          batch_max_bytes: 10485760

Permissions are like this, it was added by deployment, I didn't change it

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "logs:DescribeLogGroups",
                "s3:GetObject",
                "sqs:SendMessage"
            ],
            "Resource": [
                "arn:aws:logs:us-east-1:*****45454:log-group:/aws/lambda/condo-loan-asyc-dev-check-expired-order:*",
                "arn:aws:s3:::elk-config-sar-bucket-dev/sarconfig.yaml",
                "arn:aws:sqs:us-east-1:*****45454:elastic-serverless-forwarder-replay-queue-128e6a1cd335",
                "arn:aws:sqs:us-east-1:*****45454:elastic-serverless-forwarder-continuing-queue-128e6a1cd335"
            ]
        }
    ]
}

I did a test and released it with permission for everyone:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "sqs:SendMessage"
            ],
            "Resource": [
                "arn:aws:s3:::elk-config-sar-bucket-dev/sarconfig.yaml",
                "arn:aws:sqs:us-east-1:*****45454:elastic-serverless-forwarder-replay-queue-128e6a1cd335",
                "arn:aws:sqs:us-east-1:*****45454:elastic-serverless-forwarder-continuing-queue-128e6a1cd335"
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "logs:*",
            "Resource": [
                "arn:aws:logs:us-east-1:*****45454:log-group:/aws/lambda/pod-loan-asyc-dev-check-expired-order:*",
                "arn:aws:logs:*:*****45454:log-group:*:log-stream:*",
                "arn:aws:logs:*:*****45454:destination:*",
                "arn:aws:logs:*:*****45454:log-group:*"
            ]
        }
    ]
}

Now we have no error log, but no input yet in Elastic.

{
    "@timestamp": "2022-06-24T06:31:54.515Z",
    "log.level": "warning",
    "message": "no input defined",
    "ecs": {
        "version": "1.6.0"
    },
    "input_id": "arn:aws:logs:us-east-1:*****45454:log-group:/aws/lambda/pod-loan-asyc-dev-check-expired-order:*",
    "input_type": "cloudwatch-logs",
    "log": {
        "logger": "root",
        "origin": {
            "file": {
                "line": 119,
                "name": "handler.py"
            },
            "function": "lambda_handler"
        },
        "original": "no input defined"
    },
    "process": {
        "name": "MainProcess",
        "pid": 9,
        "thread": {
            "id": 139822395262784,
            "name": "MainThread"
        }
    }
}

this should be arn:aws:logs:us-east-1:*****45454:log-group:/aws/lambda/pod-loan-asyc-dev-check-expired-order

that's why the permission was give to the wrong resource:
arn:aws:logs:us-east-1:*****45454:log-group:/aws/lambda/condo-loan-asyc-dev-check-expired-order:*
vs
arn:aws:logs:us-east-1:*****45454:log-group:*:*

while this solves your current issue it is indeed a bug in the forwarder: the correct ARN format for cloudwatch logs log groups is the one you've provided, not the one expected by the forwarder, I will fix ASAP

Ok @Andrea_Spacca , thank you.

After the change the permission was given correctly. The permission issue is ok. But the data is still not sent, is this the BUG?

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::elk-config-sar-bucket-dev/sarconfig.yaml",
            "Effect": "Allow"
        },
        {
            "Action": "logs:DescribeLogGroups",
            "Resource": "arn:aws:logs:us-east-1:*******3299:log-group:*:*",
            "Effect": "Allow"
        },
        {
            "Action": "sqs:SendMessage",
            "Resource": [
                "arn:aws:sqs:us-east-1:*******3299:elastic-serverless-forwarder-replay-queue-128e6a1cd335",
                "arn:aws:sqs:us-east-1:*******3299:elastic-serverless-forwarder-continuing-queue-128e6a1cd335"
            ],
            "Effect": "Allow"
        }
    ]
}

This will happen because the arn is incorrect, because the correct arn has the asterisk, correct?

if you haven't changed the config file from the last you pasted it should work properly (since it includes the trailing :*):

inputs:
  - type: "cloudwatch-logs"
    id: "arn:aws:logs:us-east-1:*****45454:log-group:/aws/lambda/pod-loan-asyc-dev-check-expired-order:*"

Now it's working. But what I wanted was lambda invocation metrics, errors, memory usage, but this integration doesn't provide this data. Do I have to do anything additional?

This image from asset dashboard AWS Lambda.

I even configured this integration, but it didn't bring data.

Do I need to configure this output somewhere in sarconfig?

@Andrea_Spacca @stephenb can you help me, please?