How did u configure ur CLI? Make sure to set the default region as us-east-1 as that is what Filebeat uses by default to query the bucket location.
My bucket is in ca-central-1, which is what i specified, otherwise im getting 403...
$> AWS_DEFAULT_REGION=us-east-1 aws s3api get-bucket-location --bucket bucket-in-ca-central-1 --profile UserA
An error occurred (AccessDenied) when calling the GetBucketLocation operation: Access Denied
# not sure whats the difference between setting AWS_DEFAULT_REGION vs --region, but either way it's getting 403 when poking at us-east-1
$> aws s3api get-bucket-location --bucket bucket-in-ca-central-1 --profile UserA --region ca-central-1
{
"LocationConstraint": "ca-central-1"
}
Heres the wall of text from debug, please let me know if i should squash it to something smaller...
$> aws s3api get-bucket-location --debug --bucket bucket-in-ca-central-1 --endpoint-url https://s3.us-east-1.amazonaws.com --profile UserA
# loosely modified to fit 13k char limit
CLI version: aws-cli/2.3.4 Python/3.8.8 Windows/10 exe/AMD64
Arguments entered to CLI: ['s3api', 'get-bucket-location', '--debug', '--bucket', 'bucket-in-ca-central-1', '--endpoint-url', 'https://s3.us-east-1.amazonaws.com', '--profile', 'UserA']
Event building-command-table.main: <function add_s3 at 0x00000186E73AAA60>
Event building-command-table.main: <function add_ddb at 0x00000186E7204790>
Event building-command-table.main: <bound method BasicCommand.add_command of <class 'awscli.customizations.configure.configure.ConfigureCommand'>>
Event building-command-table.main: <function change_name at 0x00000186E71A7EE0>
Event building-command-table.main: <function change_name at 0x00000186E71B5040>
Event building-command-table.main: <function alias_opsworks_cm at 0x00000186E73BC4C0>
Event building-command-table.main: <function add_history_commands at 0x00000186E72515E0>
Event building-command-table.main: <bound method BasicCommand.add_command of <class 'awscli.customizations.devcommands.CLIDevCommand'>>
Event building-command-table.main: <function add_waiters at 0x00000186E73B3700>
Loading JSON file: C:\Program Files\Amazon\AWSCLIV2\awscli\data\cli.json
Event top-level-args-parsed: <function resolve_types at 0x00000186E73015E0>
Event top-level-args-parsed: <function no_sign_request at 0x00000186E7307160>
Event top-level-args-parsed: <function resolve_verify_ssl at 0x00000186E73070D0>
Event top-level-args-parsed: <function resolve_cli_read_timeout at 0x00000186E7307280>
Event top-level-args-parsed: <function resolve_cli_connect_timeout at 0x00000186E73071F0>
Event top-level-args-parsed: <built-in method update of dict object at 0x00000186E7458800>
Setting config variable for profile to 'UserA'
CLI version: aws-cli/2.3.4 Python/3.8.8 Windows/10 exe/AMD64 prompt/off
Arguments entered to CLI: ['s3api', 'get-bucket-location', '--debug', '--bucket', 'bucket-in-ca-central-1', '--endpoint-url', 'https://s3.us-east-1.amazonaws.com', '--profile', 'UserA']
Event session-initialized: <function add_timestamp_parser at 0x00000186E73AD0D0>
Event session-initialized: <function register_uri_param_handler at 0x00000186E6758B80>
Event session-initialized: <function add_binary_formatter at 0x00000186E741DE50>
Event session-initialized: <function no_pager_handler at 0x00000186E6753F70>
Event session-initialized: <function inject_assume_role_provider_cache at 0x00000186E6FBA820>
IMDS ENDPOINT: http://169.254.169.254/
Skipping environment variable credential check because profile name was explicitly
set.
Event session-initialized: <function attach_history_handler at 0x00000186E72514C0>
Event session-initialized: <function inject_json_file_cache at 0x00000186E71FE670>
Loading JSON file: C:\Program Files\Amazon\AWSCLIV2\awscli\botocore\data\s3\2006-03-01\service-2.json
Event building-command-table.s3api: <function add_waiters at 0x00000186E73B3700>
Loading JSON file: C:\Program Files\Amazon\AWSCLIV2\awscli\botocore\data\s3\2006-03-01\waiters-2.json
OrderedDict([('bucket', <awscli.arguments.CLIArgument object at 0x00000186E77F88E0>), ('expected-bucket-owner', <awscli.arguments.CLIArgument object at 0x00000186E77F8910>)])
Event building-argument-table.s3api.get-bucket-location: <function add_streaming_output_arg at 0x00000186E73AD670>
Event building-argument-table.s3api.get-bucket-location: <function add_cli_input_json at 0x00000186E6FC20D0>
Event building-argument-table.s3api.get-bucket-location: <function add_cli_input_yaml at 0x00000186E6FC23A0>
Event building-argument-table.s3api.get-bucket-location: <function unify_paging_params at 0x00000186E7204DC0>
Loading JSON file: C:\Program Files\Amazon\AWSCLIV2\awscli\botocore\data\s3\2006-03-01\paginators-1.json
Event building-argument-table.s3api.get-bucket-location: <function add_generate_skeleton at 0x00000186E72F7B80>
Event before-building-argument-table-parser.s3api.get-bucket-location: <bound method OverrideRequiredArgsArgument.override_required_args of <awscli.customizations.cliinput.CliInputJSONArgument object at 0x00000186E77F87C0>>
Event before-building-argument-table-parser.s3api.get-bucket-location: <bound method OverrideRequiredArgsArgument.override_required_args of <awscli.customizations.cliinput.CliInputYAMLArgument object at 0x00000186E77F89D0>>
Event before-building-argument-table-parser.s3api.get-bucket-location: <bound method GenerateCliSkeletonArgument.override_required_args of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x00000186E77F8B50>>
Event load-cli-arg.s3.get-bucket-location.bucket: <awscli.paramfile.URIArgumentHandler object at 0x00000186E749DF40>
Event process-cli-arg.s3.get-bucket-location: <awscli.argprocess.ParamShorthandParser object at 0x00000186E6F7E820>
Unpacked value of 'bucket-in-ca-central-1' for parameter "bucket": 'bucket-in-ca-central-1'
Event load-cli-arg.s3.get-bucket-location.expected-bucket-owner: <awscli.paramfile.URIArgumentHandler object at 0x00000186E749DF40>
Event load-cli-arg.s3.get-bucket-location.cli-input-json: <awscli.paramfile.URIArgumentHandler object at 0x00000186E749DF40>
Event load-cli-arg.s3.get-bucket-location.cli-input-yaml: <awscli.paramfile.URIArgumentHandler object at 0x00000186E749DF40>
Event load-cli-arg.s3.get-bucket-location.generate-cli-skeleton: <awscli.paramfile.URIArgumentHandler object at 0x00000186E749DF40>
Event calling-command.s3api.get-bucket-location: <bound method CliInputArgument.add_to_call_parameters of <awscli.customizations.cliinput.CliInputJSONArgument object at 0x00000186E77F87C0>>
Event calling-command.s3api.get-bucket-location: <bound method CliInputArgument.add_to_call_parameters of <awscli.customizations.cliinput.CliInputYAMLArgument object at 0x00000186E77F89D0>>
Event calling-command.s3api.get-bucket-location: <bound method GenerateCliSkeletonArgument.generate_skeleton of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x00000186E77F8B50>>
IMDS ENDPOINT: http://169.254.169.254/
Starting new HTTP connection (1): 169.254.169.254:80
Caught retryable HTTP exception while making metadata service request to http://169.254.169.254/latest/api/token: Could not connect to the endpoint URL: "http://169.254.169.254/latest/api/token"
OSError: [WinError 10051] A socket operation was attempted to an unreachable network
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "http://169.254.169.254/latest/api/token"
Starting new HTTP connection (2): 169.254.169.254:80
Caught retryable HTTP exception while making metadata service request to http://169.254.169.254/latest/meta-data/placement/availability-zone/: Could not connect to the endpoint URL: "http://169.254.169.254/latest/meta-data/placement/availability-zone/"
OSError: [WinError 10051] A socket operation was attempted to an unreachable network
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "http://169.254.169.254/latest/meta-data/placement/availability-zone/"
Max number of attempts exceeded (1) when attempting to retrieve data from metadata service.
Looking for credentials via: assume-role
Looking for credentials via: assume-role-with-web-identity
Looking for credentials via: sso
Looking for credentials via: shared-credentials-file
Found credentials in shared credentials file: ~/.aws/credentials
Loading JSON file: C:\Program Files\Amazon\AWSCLIV2\awscli\botocore\data\endpoints.json
Event choose-service-name: <function handle_service_name_alias at 0x00000186E5F88EE0>
Event creating-client-class.s3: <function add_generate_presigned_post at
0x00000186E5F3B700>
Event creating-client-class.s3: <function add_generate_presigned_url at 0x00000186E5F3B4C0>
Setting s3 timeout as (60, 60)
Event provide-client-params.s3.GetBucketLocation: <function base64_decode_input_blobs at 0x00000186E741F5E0>
Event before-parameter-build.s3.GetBucketLocation: <function validate_bucket_name at 0x00000186E5FAB0D0>
Event before-parameter-build.s3.GetBucketLocation: <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x00000186E7A85E50>>
Event before-parameter-build.s3.GetBucketLocation: <bound method S3ArnParamHandler.handle_arn of <botocore.utils.S3ArnParamHandler object at 0x00000186E7A85F10>>
Event before-parameter-build.s3.GetBucketLocation: <function generate_idempotent_uuid at 0x00000186E5FA8EE0>
Event before-call.s3.GetBucketLocation: <function add_expect_header at 0x00000186E5FAB430>
Event before-call.s3.GetBucketLocation: <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x00000186E7A85E50>>
Event before-call.s3.GetBucketLocation: <function inject_api_version_header_if_needed at 0x00000186E5FAF790>
Making request for OperationModel(name=GetBucketLocation) with params: {'url_path': '/bucket-in-ca-central-1?location', 'query_string': {}, 'method': 'GET', 'headers': {'User-Agent': 'aws-cli/2.3.4 Python/3.8.8 Windows/10 exe/AMD64 prompt/off command/s3api.get-bucket-location'}, 'body': b'', 'url': 'https://s3.us-east-1.amazonaws.com/bucket-in-ca-central-1?location', 'context': {'client_region': 'aws-global', 'client_config': <botocore.config.Config object at 0x00000186E7A852B0>, 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'bucket-in-ca-central-1'}}}
Event request-created.s3.GetBucketLocation: <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x00000186E7A85280>>
Event choose-signer.s3.GetBucketLocation: <bound method S3EndpointSetter.set_signer of <botocore.utils.S3EndpointSetter object at 0x00000186E7A85FA0>>
Event choose-signer.s3.GetBucketLocation: <function set_operation_specific_signer at 0x00000186E5FA8DC0>
Event before-sign.s3.GetBucketLocation: <bound method S3EndpointSetter.set_endpoint of <botocore.utils.S3EndpointSetter object at 0x00000186E7A85FA0>>
Using S3 path style addressing.
Calculating signature using v4 auth.
CanonicalRequest:
GET
/bucket-in-ca-central-1
location=
host:s3.us-east-1.amazonaws.com
x-amz-content-sha256: ...
x-amz-date: ...
StringToSign:
AWS4-HMAC-SHA256
Signature: ...
Sending http request: <AWSPreparedRequest stream_output=False, method=GET, url=https://s3.us-east-1.amazonaws.com/bucket-in-ca-central-1?location, headers={'User-Agent': b'aws-cli/2.3.4 Python/3.8.8 Windows/10 exe/AMD64 prompt/off command/s3api.get-bucket-location', 'X-Amz-Date': b''Authorization': b'AWS4-HMAC-SHA256 Credential=.../ntent-sha256;x-amz-date, Signature=...'}>
Certificate path: C:\Program Files\Amazon\AWSCLIV2\awscli\botocore\cacert.pem
Starting new HTTPS connection (1): s3.us-east-1.amazonaws.com:443
https://s3.us-east-1.amazonaws.com:443 "GET /bucket-in-ca-central-1?location HTTP/1.1" 403 None
Response headers: {'x-amz-request-id': '...', 'x-amz-id-2': '...', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Date': 'Sun, 12 Dec Response body:
b'<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>...</RequestId><HostId>...</HostId></Error>'
Event needs-retry.s3.GetBucketLocation: <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x00000186E7A85DF0>>
BUG - Not retrying request.
Event needs-retry.s3.GetBucketLocation: <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x00000186E7A85E50>>
Event after-call.s3.GetBucketLocation: <function parse_get_bucket_location at 0x00000186E5FABB80>
Event after-call.s3.GetBucketLocation: <function enhance_error_msg at 0x00000186E73AACA0>
Event after-call.s3.GetBucketLocation: <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x00000186E7A85A00>>
Exception caught in main()
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the GetBucketLocation operation: Access Denied
An error occurred (AccessDenied) when calling the GetBucketLocation operation: Access Denied
The difference is it just uses the default region if u don't specify the region like u did with the cli argument. It seems like ur issue is the exact same as the OP. Do you have any security policies that doesn't allow connections to the us-east-1 region?
Do you have any security policies that doesn't allow connections to the us-east-1 region?
Not as far as i know... I have a similar filebeat setup under the same account: UserA --> BucketA (ca-central-1) and that works. I'm now just trying to do cross account from UserA --> BucketB (ca-central-1) with identical policies to the dots and im getting stuck... 
It must be something with the security settings somewhere since the cli also has that same affect. Does it work if u use user A with bucket A using the us-east-1 region?
I would like to know how this magic is wired up...
yes strangely it works, not sure what sorcery is this... i'll keep digging...
My use case is a bit different that im trying to set up the beat for cross account bucket. I have confirmed with aws support that get-bucket-location uses us-east-1 as the default region to query for cross account buckets, and 403 is expected if the bucket is somewhere else other than us-east-1. I've also read this that it's not possible to specify a custom region in filebeat... so is there any other way around it?
$> aws s3api get-bucket-location --bucket bucketB-in-ca-central-1 --profile UserA
# this would assume us-east-1 as the default region
An error occurred (AccessDenied) when calling the GetBucketLocation operation: Access Denied
$> aws s3api get-bucket-location --bucket bucketB-in-ca-central-1 --profile UserA --region ca-central-1
THIS WORKS!
I have a fix above for the input to enable the use of a default region. I just haven't gotten around to submitting a PR for it. I'll do it later today, but it may not go into affect until 8.0. For the immediate, I don't think there is much u can do unless u build filebeat from source with the changes.
1 Like
Draft PR Add default region config to AWS by legoguy1000 · Pull Request #29415 · elastic/beats · GitHub submitted. I can build from source and send the binary to you (or you can) if u want to test it?
1 Like
@legoguy1000 PR should address this, but I found counter intuitive that a cross account bucket GetBucketLocation request fails according to the region.
could you share UserA profile anonymised permission? and also bucketB-in-ca-central-1 policy?
I will try to reproduce so that you might find a solution while waiting for the PR to be released
yup i've put it a couple of posts up right here. Looking forward to the patch as I'm taking over a bunch of buckets from 10+ accounts and trying to centralize them into a single elk.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.