The security token included in the request is invalid. (Service: AmazonDynamoDBv2)

[lvthillo]

I've setup logstash on an EC2 using an instance profile (role).
As input we have filebeat, cloudwatch and kinesis. Filebeat and cloudwatch are working fine and are proving the instance profile is working.

Also when I use AWS CLI I'm able to perform what the EC2 is allowed to do (by it's role/instance profile) and I'm not able to do what is not described in the role.

Although it seems not to work for the kinesis plugin. It was working when we were still using hardcoded accesskeys so it should not be an issue with the policies/permissions.

cat /etc/logstash/conf.d/logstash_kinesis_xxx.conf

input {
  kinesis {
    kinesis_stream_name => "xxxstream-dev"
    application_name => "xxx-logstash-dev"
    profile => "logstash"
    codec => "json_lines"
    region => "eu-west-1"
    tags => ["xxx"]
  }
}

output {
  if "auth0_log" in [tags] {
    elasticsearch {
      hosts => "https://vpc-elasticsearch-xxx:443"
      template => "/etc/logstash/template.json"
      template_overwrite => true
      template_name => "elk-stack-xxx"
      manage_template => true
      index => "xxx-log-%{+YYYY.MM.dd}"
      document_type => "xxx-log-type"
    }
  }
}

We have just removed the hardcoded keys and hoped that it would pickup the temporary credentials (like the other plugins seem to do but it does not work):

Error in syslog:
SEVERE: Failed to get table status for xxx-logstash-dev Apr 19 06:47:06 ip-10-228-113-31 logstash[3703]: com.amazonaws.services.kinesis.leases.exceptions.DependencyException: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: The security token included in the request is invalid. (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: UnrecognizedClientException; Request ID: xxx)

It seems to retry it 20 times and then terminate the kinesis pipeline.

Error in logstash log:
[2019-04-19T06:44:19,435][INFO ][logstash.agent ] Pipelines running {:count=>3, :running_pipelines=>[:filebeat, :cloudwatch, :kinesis], :non_running_pipelines=>}
[2019-04-19T06:47:27,641][INFO ][logstash.pipeline ] Pipeline has terminated {:pipeline_id=>"kinesis", :thread=>"#<Thread:0x111e76ec@/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:47 run>"}

what is wrong here?

Also this works from on the ec2:
aws dynamodb describe-table --table-name xxx-logstash-dev --region eu-west-1