Functionbeat cloudwatch S3 AccessDenied

Elastic Stack Beats
1

Following the documents, I am trying to configure functionbeat to ingest cloudwatch logs to elastic cloud deployment.

I have already given 'AmazonS3FullAccess' policy to the IAM user I use for this operation. I just tested with "aws s3api" commands from the same terminal and I could create buckets/objects. So looks like the user has sufficient access, but functionbeat is giving me errors. Not sure where I am going wrong.

./functionbeat deploy cloudwatch
Function: cloudwatch, could not deploy, error: AccessDenied: Access Denied
status code: 403, request id: C21C83CAA3DDE578, host id: Fi7PdlkGjz/dcvvgXEnguDh0xb6cS6WFN97j68WNeQPqxIZ6s4evNsOMiZbQpep63xKyvBkui04=
Fail to deploy 1 function(s)
2

Filebeat uses the credentials you set in the AWS_* environment variables to authenticate with AWS. Are you sure you set it correctly?

If those are ok, could you please share the debug logs (./functionbeat deploy cloudwatch -e -d "*")?

3

AWS_ACCESS_KEY_ID, AWS_ACCESS_KEY_ID, AWS_DEFAULT_REGION environment variables are set. aws cli uses the same credentials and was able to create s3 buckets with it.

Below is the output of ./functionbeat deploy cloudwatch -e -d "*"
(output is broken into two replies)

4 
2019-06-26T06:50:55.030-0700	INFO	instance/beat.go:571	Home path: [/Users/nivead/elk/functionbeat-7.1.1-darwin-x86_64] Config path: [/Users/nivead/elk/functionbeat-7.1.1-darwin-x86_64] Data path: [/Users/nivead/elk/functionbeat-7.1.1-darwin-x86_64/data] Logs path: [/Users/nivead/elk/functionbeat-7.1.1-darwin-x86_64/logs]
2019-06-26T06:50:55.030-0700	DEBUG	[beat]	instance/beat.go:623	Beat metadata path: /Users/nivead/elk/functionbeat-7.1.1-darwin-x86_64/data/meta.json
2019-06-26T06:50:55.031-0700	INFO	instance/beat.go:579	Beat ID: af1d1b4d-a387-4fcf-b02c-562359a38c50
2019-06-26T06:50:55.031-0700	INFO	[index-management.ilm]	ilm/ilm.go:129	Policy name: functionbeat-7.1.1
2019-06-26T06:50:55.032-0700	DEBUG	[processors]	processors/processor.go:66	Processors:
2019-06-26T06:50:55.033-0700	DEBUG	[cli-handler]	cmd/cli_handler.go:46	Starting deploy for: cloudwatch
2019-06-26T06:50:55.033-0700	DEBUG	[aws]	aws/cli_manager.go:265	Deploying function: cloudwatch
2019-06-26T06:50:55.034-0700	DEBUG	[aws]	aws/cli_manager.go:184	Compressing all assets into an artifact
2019-06-26T06:50:55.034-0700	DEBUG	[keystore]	keystore/keystore.go:89	Loading file keystore from /Users/nivead/elk/functionbeat-7.1.1-darwin-x86_64/data/functionbeat.keystore
2019-06-26T06:50:56.281-0700	DEBUG	[aws]	aws/cli_manager.go:189	Compression is successful (zip size: 17123825 bytes)
2019-06-26T06:50:56.322-0700	DEBUG	[aws]	aws/cli_manager.go:215	Using cloudformation template:
{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Resources": {
    "fnbcloudwatch": {
      "Properties": {
        "Code": {
          "S3Bucket": "nivead-elk",
          "S3Key": "functionbeat-deployment/cloudwatch/elHpaLbxjeCponOMos_Qk-mYIuS6CoHu-uueVbvdLpQ/functionbeat.zip"
        },
        "Environment": {
          "Variables": {
            "BEAT_STRICT_PERMS": "false",
            "ENABLED_FUNCTIONS": "cloudwatch"
          }
        },
        "FunctionName": "cloudwatch",
        "Handler": "functionbeat",
        "MemorySize": 128,
        "ReservedConcurrentExecutions": 5,
        "Role": {
          "Fn::GetAtt": [
            "fnbcloudwatchIAMRoleLambdaExecution",
            "Arn"
          ]
        },
        "Runtime": "go1.x",
        "Timeout": 3
      },
      "Type": "AWS::Lambda::Function"
    },
    "fnbcloudwatchIAMRoleLambdaExecution": {
      "Properties": {
        "AssumeRolePolicyDocument": {
          "Statement": [
            {
              "Action": "sts:AssumeRole",
              "Effect": "Allow",
              "Principal": {
                "Service": {
                  "Fn::Join": [
                    "",
                    [
                      "lambda.",
                      {
                        "Ref": "AWS::URLSuffix"
                      }
                    ]
                  ]
                }
              }
            }
          ]
        },
        "Path": "/",
        "Policies": [
          {
            "PolicyDocument": {
              "Statement": [
                {
                  "Action": [
                    "logs:CreateLogStream",
                    "Logs:PutLogEvents"
                  ],
                  "Effect": "Allow",
                  "Resource": [
                    {
                      "Fn::Sub": "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/cloudwatch:*"
                    }
                  ]
                }
              ]
            },
            "PolicyName": {
              "Fn::Join": [
                "-",
                [
                  "fnb",
                  "lambda",
                  "cloudwatch"
                ]
              ]
            }
          }
        ],
        "RoleName": "functionbeat-lambda-cloudwatch"
      },
      "Type": "AWS::IAM::Role"
    },
    "fnbcloudwatchLogGroup": {
      "Properties": {
        "LogGroupName": "/aws/lambda/cloudwatch"
      },
      "Type": "AWS::Logs::LogGroup"
    },
    "fnbcloudwatchPermission0": {
      "Properties": {
        "Action": "lambda:InvokeFunction",
        "FunctionName": {
          "Fn::GetAtt": [
            "fnbcloudwatch",
            "Arn"
          ]
        },
        "Principal": {
          "Fn::Join": [
            "",
            [
              "logs.",
              {
                "Ref": "AWS::Region"
              },
              ".",
              {
                "Ref": "AWS::URLSuffix"
              }
            ]
          ]
        },
        "SourceArn": {
          "Fn::Join": [
            "",
            [
              "arn:",
              {
                "Ref": "AWS::Partition"
              },
              ":logs:",
              {
                "Ref": "AWS::Region"
              },
              ":",
              {
                "Ref": "AWS::AccountId"
              },
              ":log-group:",
              "/aws/ecs/nivead-sandbox/execution",
              ":*"
            ]
          ]
        }
      },
      "Type": "AWS::Lambda::Permission"
    },
    "fnbcloudwatchSFawsecsniveadsandboxexecution": {
      "Properties": {
        "DestinationArn": {
          "Fn::GetAtt": [
            "fnbcloudwatch",
            "Arn"
          ]
        },
        "FilterPattern": "",
        "LogGroupName": "/aws/ecs/nivead-sandbox/execution"
      },
      "Type": "AWS::Logs::SubscriptionFilter"
    }
  }
}
5 
   2019-06-26T06:50:56.322-0700	DEBUG	[aws.executor]	aws/executor.go:46	The executor is executing '6' operations for converging state
    2019-06-26T06:50:56.322-0700	DEBUG	[aws]	aws/op_ensure_bucket.go:31	Verifying presence of S3 bucket: nivead-elk
    2019-06-26T06:50:57.326-0700	DEBUG	[aws]	aws/op_ensure_bucket.go:48	Could not create bucket, resp: <nil>
    2019-06-26T06:50:57.326-0700	DEBUG	[aws.executor]	aws/executor.go:68	The executor is rolling back previous execution, '0' operations to rollback
    2019-06-26T06:50:57.326-0700	DEBUG	[aws.executor]	aws/executor.go:81	The rollback is successful
    2019-06-26T06:50:57.326-0700	DEBUG	[aws]	aws/cli_manager.go:269	Deploy finish for function 'cloudwatch'
    Function: cloudwatch, could not deploy, error: AccessDenied: Access Denied
    	status code: 403, request id: A947C557B37F0BFD, host id: uWO3QdBC1gAobQmJ7GTDFArDoSfgtk3s8UpiyzDQrKhfmKFABlWdMDmRz0ub2GuSqjSWTtdnQM4=
    2019-06-26T06:50:57.327-0700	DEBUG	[cli-handler]	cmd/cli_handler.go:64	Deploy execution ended
    Fail to deploy 1 function(s)
6

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.